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Summary 

Congress  and  the  executive,  as  well  as  outside  organizations,  have  long  been  attentive  to  the 
evaluation  of  federal  programs,  with  frequent  interest  paid  to  the  independent  status  of  the 
evaluator.  This  interest  continues  into  the  current  era,  with  numerous  illustrations  of  the  multi¬ 
faceted  approaches  adopted  and  proposed. 

An  evaluation  may  provide  information  at  any  stage  of  the  policy  process  about  how  a  federal 
government  policy,  program,  activity,  or  agency  is  working.  Congress  has  required  evaluations 
through  legislation  (or  requested  these  via  its  committee  and  Member  offices);  and  the  executive 
branch  has  pursued  evaluations  through  presidential  or  agency  directives. 

Part  of  choosing  how  to  carry  out  an  evaluation  involves  deciding  if  some  kind  of 
“independence”  would  be  a  desirable  attribute.  Observers  often  see  independence  as  a  means  of 
avoiding  or  deterring  bias  and  ensuring  an  objective,  impartial  assessment.  In  the  context  of 
evaluation,  independence  may  apply  to  an  evaluation  or  to  an  evaluator.  On  one  hand,  for 
example,  the  term  may  relate  to  independence  of  an  evaluation  from  the  policy  preferences  of  an 
individual  or  group  (“independent  evaluation”),  perhaps  by  prohibiting  political  appointees  from 
revising  or  evaluating  a  program.  Independence  may  refer  to  an  entity  that  conducts  evaluations 
that  also  is  located  outside  the  immediate  organization  responsible  for  policy  implementation 
(“independent  evaluator”). 

There  is  some  diversity  of  opinion  regarding  the  definition  of  independence  and  how  it  might  be 
ensured.  For  example,  an  evaluator’s  “external”  status,  outside  the  organization  that  is 
implementing  a  program,  does  not  necessarily  equate  with  independence.  Nor  would  an 
evaluator’s  “internal”  status,  inside  the  implementing  organization,  necessarily  equate  with  a  lack 
of  independence  for  an  evaluation  (e.g.,  if  an  expert  panel  reviewed  the  internally  produced 
evaluation  for  bias).  There  is  varying  opinion  concerning  when  independence  is  necessary,  or 
possibly  counterproductive,  and  what  value  it  may  bring. 

The  differences  of  opinion  among  definitions  and  perceived  need  notwithstanding,  instances  of 
independent  program  evaluators  appear  to  be  growing  in  number  and  variety  at  the  federal  level. 

This  report  focuses  on  examples  of  independent  evaluators  (IEs):  when  an  evaluation  is  to  be 
conducted  by  an  entity  outside  the  immediate  organization  that  is  responsible  for  policy 
implementation,  and  the  entity  also  is  intended  to  have  one  or  more  dimensions  of  independence. 
IEs  and  similar  constructs,  however,  vary  across  a  number  of  characteristics  and  attributes: 
structure,  jurisdiction,  authority,  resources,  length  of  tenure,  and  specific  duties  and 
responsibilities.  These  differences,  in  turn,  could  affect  their  capabilities,  effectiveness,  and 
assistance  to  others,  including  their  contributions  to  the  oversight  of  a  program  or  project  by 
Congress  and  the  executive  branch. 

After  an  overview  of  such  entities — which  encompass  new  units  created  specifically  for 
conducting  an  evaluation  as  well  as  existing  ones,  such  as  the  Government  Accountability  Office 
and  offices  of  the  inspectors  general — this  report  suggests  possible  broad  characteristics  and 
criteria  of  independent  evaluators  or  similar  units,  which  could  be  valuable  in  oversight  or 
legislative  endeavors.  The  Appendix  describes  a  number  of  such  offices — past,  present,  and 
proposed — along  with  citations  to  relevant  official  documents  and  other  materials  for  each 
example  (public  laws,  legislative  proposals,  executive  branch  directives,  and  secondary  analyses). 
This  report  will  be  updated  as  conditions  warrant. 
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Introduction 

An  evaluation  may  provide  information  at  any  stage  of  the  policy  process  about  how  a  federal 
government  policy,  program,  activity,  or  agency  is  working.  The  resulting  information  may  be 
used  to  better  understand  policy  problems  and  to  help  inform  the  design,  implementation,  and 
oversight  of  policies.  A  variety  of  stakeholders  may  seek  evaluation  information  and  data  to  help 
them  exercise  their  responsibilities,  obligations,  and  rights  in  a  representative  democracy, 
including  the  establishment,  implementation,  oversight,  and  other  requirements  of  a  federal 
program  or  project.  These  stakeholders  and  interested  parties  include,  among  others,  Congress, 
the  President,  federal  agency  officers  and  employees,  state  and  local  governments,  interest 
groups,  private  sector  contractors,  the  news  media,  and  the  general  public. 

For  a  number  of  reasons,  there  is  not  necessarily  a  best  way  to  design  and  carry  out  an  evaluation 
of  a  particular  program,  activity,  or  operation.  A  program’s  context,  including  its  longevity  and 
the  manner  in  which  it  is  implemented,  typically  influences  choices  about  evaluation  design.  In 
addition,  stakeholders  often  have  different  needs  and  research  questions  about  which  they  would 
like  more  information. 1  One  aspect  of  choosing  how  to  carry  out  an  evaluation  involves  deciding 
when  some  kind  of  “independence”  would  be  a  desirable  attribute. 

In  the  context  of  evaluation,  independence  may  apply  to  an  evaluation  or  to  an  evaluator.2  On  one 
hand,  for  example,  the  term  may  relate  to  independence  of  an  evaluation  from  the  policy 
preferences  of  an  individual  or  group  (“independent  evaluation”),  perhaps  by  prohibiting  political 
appointees  from  revising  an  evaluation.  Independence  also  may  refer  to  an  entity  that  conducts 
evaluations  that  also  is  located  outside  the  immediate  organization  responsible  for  policy 
implementation  (“independent  evaluator”). 

There  is  some  diversity  of  opinion  regarding  how  to  define  independence  and  what  makes  an 
evaluator  (or  evaluation)  independent.  For  example,  an  evaluator’s  “external”  status,  outside  the 
organization  that  is  implementing  a  program,  does  not  necessarily  equate  with  independence.  Nor 
would  an  evaluator’s  “internal”  status,  inside  the  implementing  organization,  necessarily  equate 
with  a  lack  of  independence  for  an  evaluation  (e.g.,  if  an  expert  panel  reviewed  the  internally 
produced  evaluation  for  bias).  There  is  also  varying  opinion  concerning  when  independence  is 
necessary,  or  possibly  counterproductive,  and  what  value  it  may  bring.  In  some  situations,  there 
may  be  little  need  for  independence  in  evaluations  (e.g.,  when  equipping  a  program  with  capacity 
to  improve  its  own  operations  iteratively,  or  when  “adaptively”  managing  a  project  to  help  restore 
an  ecosystem).3  Nonetheless,  instances  of  independent  evaluators  appear  to  be  growing  in  number 
and  variety. 

This  report  focuses  on  the  characteristics  of  independent  evaluators  (IEs) — examples  of  which  are 
described  in  the  Appendix — when  an  evaluation  is  to  be  conducted  by  an  entity  outside  the 


1  For  discussion,  see  CRS  Report  RL33301,  Congress  and  Program  Evaluation:  An  Overview  of  Randomized 
Controlled  Trials  (RCTs)  and  Related  Issues,  by  Clinton  T.  Brass,  Erin  D.  Williams,  and  Bias  Nunez-Neto. 

2  The  Library  of  Congress’s  catalogues  list  a  number  of  titles  using  “independent  evaluation’’  and  “independent 
evaluator.”  But  these  pertain  mostly  to  reports  on  and  by  such  units  in  ad  hoc,  highly  specialized  efforts,  undertaken  by 
foreign  governments,  international  organizations,  and  federal,  state,  and  local  governments  in  the  United  States. 

3  See  discussion  in  Eleanor  Chelimsky,  “The  Coming  Transformations  in  Evaluation,”  in  Eleanor  Chelimsky  and 
William  R.  Shardish,  eds..  Evaluation  for  the  21st  Century:  A  Handbook  (Thousand  Oaks,  CA:  Sage,  1997),  pp.  18-22. 
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immediate  organization  that  is  responsible  for  policy  implementation,  and  the  IE  is  also  intended 
to  have  one  or  more  dimensions  of  independence.  IEs  and  similar  constructs  that  conduct 
independent  evaluations,  however,  vary  across  a  number  of  attributes:  structure,  jurisdiction, 
authority,  resources,  length  of  tenure,  and  specific  duties  and  responsibilities.  These  differences, 
in  turn,  could  affect  their  capabilities,  effectiveness,  and  assistance  to  others,  including  their 
contributions  to  the  oversight  of  a  program  or  project  by  the  executive  or  legislature. 

After  an  overview  of  such  entities — which  include  newly  and  specially  created  units  as  well  as 
existing  ones,  such  as  the  Government  Accountability  Office  (GAO)  and  offices  of  inspector 
general  (OIGs) — this  report  suggests  possible  broad  characteristics  and  criteria  of  independent 
evaluators  or  similar  units.  In  situations  when  some  extent  of  independence  in  an  existing  or 
proposed  evaluator  were  deemed  desirable,  Congress  might  consider  the  different  kinds  and 
degrees  of  independence  that  could  be  pursued.  The  final  section  describes  a  number  of  such 
offices,  along  with  citations  to  relevant  materials  for  each  example  (public  laws,  legislative 
proposals,  executive  branch  documents,  and  secondary  analyses). 


Overview  of  Independent  Evaluators  and  Similar 
Units 


Evaluation  in  the  Federal  Government 

Evaluation’s  roots  have  been  characterized  as  reaching  as  far  back  as  hundreds  of  years  or,  even 
more  remarkably,  thousands  of  years.4  This  lengthy  heritage  notwithstanding,  some  observers 
describe  the  “modern  era”  of  evaluation,  and  particularly  “program  evaluation”  (focusing  on 
government  social  programs),  as  emerging  in  the  1960s  and  growing  appreciably  since  then.5 
Expansion  of  social,  economic,  and  environmental  programs,  among  others,  in  many  cases  has 
been  accompanied  by  legislation  mandating  and  funding  evaluation  or  by  executive  directives 
ordering  such  studies.  These  changes  facilitated  and  increased  the  diversification  of  program 
evaluation,  based  on  a  perceived  need  to  better  inform  the  understanding  of  policy  problems, 
formulate  responses,  and  strengthen  oversight  by  the  executive  and  legislature  over  a  growing 
expanse  and  complexity  of  federal  programs.  Accompanying  this  development,  program 
evaluation  has  also  become  more  sophisticated  and  selective  in  terms  of  its  methods, 
methodology,  type  and  level  of  operation,  and  subject  or  policy  area.6 


4  William  R.  Shadish  and  Jason  K.  Luellen,  “History  of  Evaluation,’’  in  Sandra  Mathison,  ed„  Encyclopedia  of 
Evaluation  (Thousand  Oaks,  CA:  Sage,  2005),  p.  183. 

5  William  R.  Shadish  Jr.,  Thomas  D.  Cook,  and  Laura  C.  Leviton,  Foundations  of  Program  Evaluation:  Theories  of 
Practice  (Newbury  Park,  CA:  Sage,  1991),  p.  22.  For  more  on  evaluation  and  related  concepts,  see  CRS  Report 
RL30240.  Congressional  Ox’ersight  Manual,  by  Frederick  M.  Kaiser,  et  al. ;  CRS  Report  RL33301 ,  Congress  and 
Program  Evaluation:  An  Overview  of  Randomized  Controlled  Trials  (RCTs)  and  Related  Issues,  by  Clinton  T.  Brass, 
Erin  D.  Williams,  and  Bias  Nunez-Neto;  and  U.S.  Congress,  Senate  Committee  on  Government  Operations,  Legislative 
Oversight  and  Program  Evaluation:  A  Seminar  Sponsored  by  the  Congressional  Research  Sendee,  committee  print. 

94lh  Cong.,  2nd  sess.  (Washington:  GPO,  1976).  A  more  recent  manifestation  of  program  evaluation’s  anticipated 
benefits  appeared  in  the  President’s  FY201 1  budget:  U.S.  Executive  Office  of  the  President,  Office  of  Management  and 
Budget  (hereafter  OMB),  Budget  of  the  U.S.  Government,  Fiscal  Year  2011,  Analytical  Perspectives  (Washington, 
GPO.  2010),  pp.  91-92.). 

6  Illustrative  of  this  is  U.S.  Department  of  Energy  (DOE),  Overview  of  Evaluation  Methods  for  R&D  Programs:  A 
Directory  of  Evaluation  Methods  Relevant  to  Technology  Development  Programs  (Washington.  DOE,  2007). 


Congressional  Research  Service 


2 


Independent  Evaluators  of  Federal  Programs:  Approaches,  Devices,  and  Examples 


The  1990s  gave  added  impetus  to  assessments,  evaluations,  and  reviews  of  programs,  activities, 
and  operations  through  a  number  of  laws,  which  remain  on  the  books.  Among  these  supporting 
evaluation  efforts,  directly  or  indirectly,  are  the  Chief  Financial  Officers  Act  of  1990  (CFO  Act; 
P.L.  101-576,  104  Stat.  2838);  Government  Performance  and  Results  Act  of  1993  (GPRA;  PL. 
103-62,  107  Stat.  285);  Information  Technology  Management  Reform  Act  of  1996,  later  renamed 
the  Clinger-Cohen  Act  (CCA;  Division  E  of  P.L.  104-106,  110  Stat.  679,  and  P.L.  104-208,  110 
Stat.  3009-393);  and  Federal  Financial  Assistance  Improvement  Act  of  1999  (P.L.  106-107,  113 
Stat.  I486).7  These  and  other  earlier  relevant  statutes,  including  the  Inspector  General  Act  of  1978 
(IG  Act),  as  amended,8  allow  for  flexibility  in  determining  various  evaluation  requirements, 
particularly  concerning  individual  projects  and  programs.  GPRA,  for  example,  defines  “program 
evaluation”  quite  generally  as  “an  assessment,  through  objective  measurement  and  systematic 
analysis,  of  the  manner  and  extent  to  which  Federal  programs  achieve  intended  objectives.”9 


7  Coverage  of  these  and  many  other  statutes  is  in  CRS  Reports  CRS  Report  RL30795,  General  Management  Laws:  A 
Compendium ,  by  Clinton  T.  Brass,  et  al.,  pp.  72-76. 

8  The  IG  Act  (5  U.S.C.  Appendix) — establishing  offices  of  inspectors  general  initially  in  1978 — is  among  other  earlier 
enactments  which  allow  for  some  type  of  program  evaluation  but  which  may  not  have  been  pursued  extensively  until 
the  1990s  or  later.  Until  then,  few  of  the  offices  of  inspector  general  (OIGs)  conducted  inspections/evaluations  (I&E), 
despite  implied  authority  to  do  so.  A  1990  study,  for  instance,  found  that  only  eight  of  the  more  than  60  OIGs  at  the 
time  had  an  inspection  activity  similar  to  program  evaluation,  although  its  performance  was  viewed  as  a  “a  definite  and 
increasing  trend.”  Michael  Hendricks,  Michael  F.  Mangano,  and  William  C.  Moran,  eds.,  Inspectors  General:  A  New 
Force  in  Evaluation  (San  Francisco.  Jossey-Bass  Inc.,  1990),  pp.  1  and  77.  By  2007,  the  number  had  increased 
substantially,  according  to  a  survey  of  the  57  offices  directly  under  the  IG  Act  at  the  time  (36  of  which  responded).  A 
majority  of  the  offices — 31 — reported  that  they  conducted  I&E  work.  Among  these,  25  had  staff  dedicated  to  I&E, 
either  in  stand-alone  units  or  within  other  units  (e.g.,  audit);  and  two  OIGs  had  set  up  multiple  I&E  units.  PCIE/ECIE, 
2007  Survey  of  Inspection  and  Evaluation  Units  in  the  Federal  Inspector  General  Community ,  pp.  i  and  1,  at 
http://www.ignet.gov.  Other  comparisons  further  indicate  the  expansion  of  inspection  and  evaluation  within  the  IG 
community  over  the  recent  past.  From  1999  to  2007,  the  number  of  separate  I&E  units  grew  from  15  to  25;  the  average 
annual  budget  for  them  rose  from  $2.42  million  to  $3.95  million;  and  the  number  of  full-time  equivalent  employees 
increased  from  415  to  648.  Ibid.,  p.  i.  The  belated  adoption  of  inspection  and  evaluation  in  the  IG  community  is 
reflected  in  other  developments.  It  was  not  until  1993,  importantly,  that  the  coordinative  councils  of  the  inspectors 
general  issued  the  first  set  of  quality  standards  for  inspections.  PCIE/ECIE,  Quality  Standards  for  Inspections  (revised), 
January  2005,  cover  letter,  at  http://www.ignet.gov.  At  about  the  same  time,  the  IGs  approved  a  vision  statement  that 
reflected  a  broad  change  along  the  lines  recommended  by  President  William  Clinton’s  National  Performance  Review 
(NPR),  suggesting  an  expanded  role  for  program  evaluation.  A  Government  Accountability  Office  (then  General 
Accounting  Office;  hereafter  GAO)  study  summarized  the  NPR  recommendation  for  the  IGs  to  change  “the  focus  of 
IGs  from  compliance  auditing  to  evaluating  management  control  systems  and  recasting  their  method  of  operations  to  be 
more  collaborative  and  less  adversarial.  In  January  1994,  the  IGs  adopted  an  ‘Inspectors  General  Vision  Statement’  that 
states:  “We  are  agents  of  positive  change  striving  for  continuous  improvement  in  our  agencies’  management  and 
program  operations  and  in  our  own  offices.”  GAO,  Federal  Inspectors  General:  An  Historical  Perspective,  GAO/T- 
AIMD-98-146,  April  21,  1998,  p.  4.  The  original  IG  Act  of  1978,  however,  directed  the  statutory  IGs  to  appoint 
assistant  IGs  only  for  audit  and  for  investigation  and  not  for  inspection  or  program  evaluation.  A  provision  for  all 
statutory  IGs  to  do  so  in  their  offices  has  not  been  added  by  subsequent  amendments.  Nonetheless,  the  2008 
amendments  to  the  IG  act  call  on  a  new  Council  of  the  Inspectors  General  for  Integrity  and  Efficiency  (CIGIE), 
composed  of  the  statutory  IGs  and  other  relevant  offices,  to  “develop  plans  for  coordinated,  Governmentwide  activities 
that  ...  promote  economy  and  efficiency  in  Federal  programs  and  operations,  including  interagency  and  interentity 
audit,  investigation,  inspection,  and  evaluation  programs  and  projects  ...  that  exceed  the  capability  or  jurisdiction  of  an 
individual  agency  or  entity  ”  (Inspector  General  Act  Reform  Act  of  2008,  P.L.  1 10-409,  122  Stat.  4308).  The  same 
enactment  increases  the  prominence  of  program  evaluations  by  requiring  that  the  IG  semiannual  reports  include 
“inspection  reports,  and  evaluation  reports”  (122  Stat.  4315). 

9  GPRA  or  the  Results  Act  provides  that  each  agency  develop  a  mission  statement,  on  which  is  based  a  five-year 
strategic  plan  setting  long-range  goals,  which  can  be  adjusted  to  meet  new  demands  and  changing  conditions;  an  annual 
plan  for  putting  the  goals  into  effect,  and  a  yearly  follow-up  evaluation.  The  Results  Act  is  designed  to  recognize  the 
importance  of  outcomes  (that  is,  the  measured  impact  and  effectiveness  of  a  program)  as  opposed  to  outputs  (that  is,  for 
example,  the  amount  of  money  funding  a  project,  the  issuance  of  new  regulations,  or  the  number  of  grants).  A  bill 
(continued...) 
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Independent  Evaluators  and  Similar  Units 

As  noted  earlier,  independent  evaluation  oftentimes  refers  to  the  review  and  assessment  of  how 
well  programs  and  projects  are  working  that  is  conducted  by  a  unit  outside  the  program  office 
itself.  Such  units  have  been  expressly  authorized  to  perform  independent  evaluations  or  have 
implied  authority  to  do  so  under  a  broad  mandate  to  oversee  or  review  a  program,  project, 
activity,  or  operation. 

A  Variety  of  Names  and  Titles 

Independent  evaluation  can  be,  and  has  been,  carried  out  by  positions  under  a  number  of  different 
names  and  titles.  Specific  ones  identified  in  this  study  include  independent  evaluator,  independent 
auditor,  interagency  coordinating  research  institute,  accountability  board,  peer  reviewer, 
independent  or  peer  review  panel,  program  evaluator,  program  evaluation  unit,  research  and 
evaluation  institute,  inspection  and  evaluation  (I&E)  unit  in  an  office  of  inspector  general 
(OIG),10  and  various  organizational  groupings  in  the  Government  Accountability  Office. 11  These 
have  been  established  within  the  parent  agencies  (but  separate  from  the  program  office)  or  in 
outside  organizations,  public  or  private.  In  addition,  such  entities,  whatever  their  name  and 
wherever  located,  appear  to  be  growing  in  number  and  in  a  variety  of  policy  domains. 

Lack  of  a  Precise,  Agreed-Upon  Definition 

The  concepts  of  independent  evaluator  and  independent  evaluation,  however,  have  led  to 
generalized  understandings,  rather  than  a  precise,  detailed,  agreed-upon  definition.  The  broad 
notions  lack  specification,  standardization,  and  uniformity.  There  appears  to  be  no  express, 
across-the-board  definition  of  an  “independent  evaluation”  or  “independent  evaluator”  in  federal 


(...continued) 

introduced  in  2010 — the  Government  Efficiency,  Effectiveness,  and  Performance  Act  of  2010  (H.R.  2142,  1 1 1 lh  Cong., 
2nd  sess.) — would  build  on  GPRA  in  several  ways.  One  provision,  for  instance,  would  require  quarterly  performance 
assessments  of  government  programs  in  order  to  evaluate  their  performance  and  improvement.  See  introductory 
remarks  by  Representatives  Chaffetz.  Cuellar,  Platts,  and  Watson,  in  Congressional  Record ,  June  16,  2010,  vol.  156, 
pp.  H4553-H4560;  and  CRS  congressional  distribution  memorandum.  Analysis  of  Subcommittee-Reported  H.R.  2142 
(111th  Congress)  and  Related  Issues,  by  Clinton  T.  Brass  (available  from  author). 

10  Importantly  under  the  Inspector  General  Act  of  1978,  OIGs  are  intended  to  be  independent  units,  even  though  each  is 
located  within  a  parent  "establishment'’  or  “designated  federal  entity.”  Supporting  their  independence,  inspectors 
general  operate  only  under  the  “general  supervision”  of  the  agency  head;  and  except  in  a  few  cases  and  under  specified 
circumstances,  the  head  cannot  prevent  or  halt  an  IG  investigation  or  audit  and  must  transmit  IG  reports,  including 
recommendations  for  corrective  action,  to  Congress  unaltered.  In  addition,  IGs  are  prohibited  from  undertaking 
program  operating  responsibilities.  Other  supports  include  notification  to  Congress  of  their  initial  budget  requests, 
along  with  any  changes  made  by  agency  administration  or  OMB,  and  an  IG  statement  if  he  or  she  considers  any 
reductions  to  be  harmful  to  carrying  out  the  OIG  responsibilities.  Finally,  IGs  in  establishments  are  appointed  by  the 
President  with  Senate  confirmation  and  in  DFEs  by  the  agency  head.  In  both  types  of  offices.  Congress  must  be 
notified  in  writing  of  the  reasons  30  days  in  advance  of  any  IG’s  removal  by  the  President  or  agency  head,  respectively. 
See  also  footnote  19  below. 

11  The  independence  of  GAO,  a  congressional  support  agency,  from  the  executive  agencies  it  reviews,  audits, 
investigates,  or  inspects  has  been  an  integral  part  of  the  Office  since  its  establishing  authority  in  1921.  Then  known  as 
the  General  Accounting  Office,  it  "shall  be  independent  of  the  executive  departments  and  under  the  control  and 
direction  of  the  Comptroller  General  of  the  United  States”  (P.L.  67-13,  codified  at  31  U.S.C.  41).  See  also  footnote  18 
below. 
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statute  or  regulation. 12  (Further  discussion  of  these  concepts  appears  below.)  As  a  result,  there  are 
significant  variations  among  offices  which  may  fall  under  the  same  rubric  or  which  conduct  the 
same  function.  In  other  words,  “independent,”  “evaluator,”  and  “evaluation”  are  subject  to 
different  meanings  among  offices  that  use  similar  terminology. 

Considerable  Diversity  in  Attributes  and  Characteristics 

The  research  for  this  analysis — which  identifies  a  number  of  independent  evaluators  or  similar 
entities  and  functions — found  that  none  of  the  entities  were  identical.  Although  there  were 
similarities  governing  several  key  characteristics  in  some  cases,  there  were  substantial  differences 
of  structure,  organization,  authority,  jurisdiction,  funding,  staffing,  length  of  tenure,  or  duties  and 
responsibilities. 

Dissimilarities  among  these  characteristics  in  IE-like  positions  and  functions,  in  fact,  appear  to  be 
more  common  than  their  similarities.  This  arises,  in  part,  because  of  different  rationales, 
expectations,  research  questions,  and  conditions  surrounding  the  establishment  of  the  positions. 
Because  of  this,  the  resulting  independent  evaluators  lack  standardization  and  uniformity  across- 
the-board,  and  a  number  have  been  given  substantial  flexibility  in  organizing  their  own  operation. 
Structural  differences  might  also  arise,  at  least  in  some  cases,  because  the  creator  of  the  entity 
(the  President,  agency  head,  or  Congress)  intends  to  allow  for  discretion  and  flexibility  at  the 
implementing  level  (for  the  agency  or  IE,  reflecting  his  or  her  expertise  and  experience).  Along 
with  this,  the  establishing  authority  for  many  positions — public  law,  executive  order,  or 
administrative  fiat — often  do  not  specify  certain  characteristics  of  the  office,  such  as  its  funding, 
reporting  requirements,  tenure,  or  evaluation  standards. 

Disagreements,  moreover,  have  arisen  over  the  value,  importance,  and  means  of  supporting 
independence  for  evaluators.  The  Government  Accountability  Office  (GAO),  when  it  examined 
the  Office  of  Management  and  Budget’s  (OMB ’s)  guidance  for  the  Program  Assessment  Rating 
Tool  (PART)  in  2005,  found  that  a 

source  of  tension  between  OMB  and  agency  evaluation  interests  was  the  evaluation’s 
independence.  PART  guidance  stressed  that  for  evaluations  to  be  independent,  nonbiased 
parties  with  no  conflict  of  interest,  for  example,  GAO  or  an  Inspector  General  should 
conduct  them  ....  OMB  subsequently  revised  the  guidance  to  allow  evaluations  to  be 
considered  independent  if  the  program  contracted  them  out  to  a  third  party  or  they  were 
carried  out  by  an  agency’ s  program  evaluation  office.  However,  disagreements  continued  on 
the  value  and  importance  of  this  criterion. 1 

Furthermore,  an  evaluation  scholar  noted  that  independence  may  not  be  an  important  dimension 
of  an  evaluation  if  the  objectives  of  the  evaluation  primarily  focus  on  strengthening  institutions  or 
building  “agency  or  organizational  capacity  in  some  evaluative  area.”14 


12  This  absence  is  in  contrast,  for  instance,  to  statutory  IGs  under  the  IG  Act.  Among  other  things,  the  act  spells  out  key 
commonalities,  including  their  appointment,  removal,  supervision,  broad  responsibilities  and  duties,  investigative  and 
audit  powers,  reporting  requirements,  staffing  and  funding,  and  jurisdiction.  Differences  among  IGs  under  the  IG  Act. 
which  are  relatively  few,  are  specified  in  the  law  for  an  individual  office  or  are  confined  to  a  specific  policy  matter  or 
project. 

13  GAO.  Program  Evaluation:  OMB's  PART  Reviews  Increased  Agencies’  Attention  to  Improving  Evidence  of 
Program  Results,  GAO-06-76,  October  2005,  p.  25.  Further  discussion  of  OMB  and  PART  appears  below. 

14  Eleanor  Chelimsky,  “The  Coming  Transformations  in  Evaluation,”  in  Eleanor  Chelimsky  and  William  R.  Shardish, 
(continued...) 
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Other  considerations  tie  into  program  evaluation  in  general,  no  matter  where  it  is  carried  out. 
These  could  extend  to  whether  the  subject  of  the  evaluation  is  either  (1)  too  large  and  broad  an 
undertaking  for  the  evaluator,  thereby  overwhelming  him  or  her;  or  (2)  to  the  contrary,  too  small 
and  narrowly  focused  to  be  of  utility  apart  from  the  immediate  project  (in  developing,  for 
instance,  recommendations  for  best  practices  or  the  entire  program).  Along  the  same  lines  are 
concerns  about  whether  the  evaluation  is  integrated  into  the  overall  operation  of  a  program  and 
whether  an  evaluator  might  be  perceived  as  “usurping”  or  competing  with  the  prerogatives  of  a 
parallel  office.  Others,  discussed  further  below,  deal  with  selecting  the  most  appropriate 
evaluation  technique  or  method,  especially  in  light  of  the  diversity  among  them,  as  well  as  the 
right  evaluator. 

Independent  evaluation  entities  do  not  arise  or  exist  in  a  vacuum.  Their  establishment,  powers, 
performance,  effectiveness,  and  impact  are  subject  to  a  variety  of  influences.  And  because  they 
are  usually  ad  hoc  and  idiosyncratic,  they  differ  in  their  structure  and  operation.  IE-like  offices 
range  from  modest  efforts — beginning  with  a  position  without  separate  funding  or  staffing, 
evaluating  only  a  single,  short-term  project — to  major  undertakings,  extending  to  a  separate  unit 
composed  of  a  number  of  individuals,  with  its  own  budget  and  resources,  operating  continuously 
throughout  the  life  of  a  long-term,  interagency,  and  sometimes  intergovernmental  program.  In 
between  these  ends  of  the  spectrum  are  a  number  of  possible  combinations.  Variations  exist  even 
among  units  evaluating  programs  in  the  same  broad  subject  area.  Six  of  the  examples  below,  for 
instance,  deal  with  interagency,  intergovernmental,  and,  in  some  cases,  interstate  waterway 
programs;  yet  the  evaluation  units  differ  from  one  another,  sometimes  significantly,  in  their  main 
characteristics  or  in  the  specifications  in  their  establishing  authority. 

In  sum,  the  characteristics  of  independent  evaluators  and  similar  constructs  are  determined  by  a 
number  of  formal  and  informal  factors.  Differences  among  evaluation  constructs  are  reflected  in 
the  powers  and  protections  in  the  authorities  that  established  the  IEs — public  laws,  executive 
orders,  or  administrative  directives — as  well  as  in  their  relationships  with  executive  officials. 
Besides  the  variables  already  identified,  other  influences,  some  informal  or  intangible,  add  to  this 
mixture.  These  include  the  expertise  of  the  staff,  competency  and  impartiality  of  the  evaluator, 
and  trust  and  confidence  between  an  evaluator  and  the  program  office.  These,  in  turn,  affect  the 
IEs’  actual  and  potential  independence,  capacity,  capability,  and  effectiveness — including  their 
contributions  to  oversight  by  Congress  and  the  executive — extending  from  the  immediate 
program  office  to  the  parent  agency,  OMB,  and  the  President. 


Possible  Characteristics  of  Independent  Evaluation 
Units 

As  emphasized  above,  the  terms  “independent  evaluator”  and  “independent  evaluation”  lack 
precise,  standardized,  agreed-upon  definitions  in  public  law  or  executive  directive,  and  they  vary 
considerably  across  a  number  of  dimensions.  Nonetheless,  many  possible  criteria  and  attributes  of 
an  independent  evaluator  or  similar  construct  can  be  identified,  based  on  the  research  for  this 
examination  and  on  other  sources,  both  public  and  private.  These  sources  include  the  American 


(...continued) 

eds..  Evaluation  for  the  21st  Century:  A  Handbook  (Thousand  Oaks,  CA:  Sage,  1997),  pp.  18-22. 
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Evaluation  Association  (AEA);15  evaluation  reference  works,  such  as  the  Encyclopedia  of 
Evaluation ;16  the  Joint  Committee  on  Standards  for  Educational  Evaluation  (Joint  Committee);17 
and  federal  agencies  that  conduct  and  direct  evaluations  of  programs,  projects,  activities,  and 
operations.  Illustrative  agencies  include  the  Government  Accountability  Office  (GAO),18  the 
Council  of  the  Inspectors  General  on  Integrity  and  Efficiency  (CIGIE)  and  its  predecessors,19  and 
the  Office  of  Management  and  Budget  (OMB  ). 20 


15  AEA  provides  guidance  for  scholars  and  practitioners  in  conducting  evaluations,  including  their  potential 
contributions  to  policymaking  and  program  implementation.  AEA,  Evaluation,  Evaluators,  and  the  American 
Evaluation  Association,  at  http://www.eval.org/aea09.eptf.policy.handouts.pdf;  and  American  Evaluation  Association 
Guiding  Principles  for  Evaluators  (2004),  at  http://www.eval.org/GPTraining/GP%20Training%20Final/ 
gp.principles.pdf. 

16  For  example,  see  Gail  V.  Barrington,  “Independent  Evaluation,”  in  Encyclopedia  of  Evaluation  (Thousand  Oaks, 

CA:  Sage,  2005),  p.  199. 

17  Joint  Committee  on  Standards  for  Educational  Evaluation,  The  Program  Evaluation  Standards:  How  to  Assess 
Evaluations  in  Educational  Programs  (Thousand  Oaks,  CA:  Sage,  1994).  Established  in  1975,  the  Joint  Committee  is 
comprised  of  16  professional  educational  associations,  ranging  from  the  American  Association  of  School 
Administrators,  AEA,  and  the  American  Psychological  Association  to  the  National  Council  on  Measurement  in 
Education  and  the  National  Education  Association.  The  Joint  Committee’s  experience  in  the  field  of  program 
evaluation,  including  its  recommended  standards,  has  focused  on  education  programs;  but  these  are  seen  as  also  having 
general  applicability  to  other  public  policy  areas.  See  Daniel  L.  Stufflebeam,  “Joint  Standards  for  Education 
Evaluation,”  in  Encyclopedia  of  Evaluation,  pp.  213-214. 

18  Among  the  many  GAO  products  that  demonstrate  its  long-standing  interest  in  program  evaluation  are:  Performance 
Measurement  and  Evaluation:  Definitions  and  Relationships,  GAO-05-739SP.  May  2005;  Program  Evaluation: 

OMB 's  PART  Reviews  Increased  Agencies  ’  Attention  to  Improving  Evidence  of  Program  Results,  GAO-06-67.  October 
2005;  Program  Evaluation:  An  Evaluation  Culture  and  Collaborative  Partnerships  Help  Build  Agency  Capacity, 
GAO-03-454.  May  2003;  Designing  Evaluations,  GAO/PEMD-IO.1.4.  March  1991;  Case  Study  Evaluations,  GAO- 
PEMD-91-10.1.9,  November  1990;  Prospective  Evaluation  Methods:  The  Prospective  Evaluation  Synthesis,  PEMD- 
10.1.10,  November  1,  1990;  and  The  Evaluation  Synthesis,  PEMD-10.1.2,  March  1,  1990.  Bolstering  GAO’s 
involvement  in  this  field  is  the  1982  codification  of  title  31  of  the  U.S.  Code,  Money  and  Finance,  which  provides  for 
evaluating  programs  and  activities  of  the  United  States  Government  (P.L.  97-258,  96  Stat.  893-894).  Under  it.  the 
Comptroller  General  is  to  “develop  and  recommend  to  Congress  ways  to  evaluate  a  program  or  activity”  as  well  as  to 
assist  a  requesting  congressional  committee  to  “develop  a  statement  of  legislative  goals  and  ways  to  assess  and  report 
program  performance  related  to  the  goals”  and  to  “assess  program  evaluations  prepared  by  and  for  an  agency.”  Ibid. 

19  As  noted  above,  IGs,  operating  directly  under  the  Inspector  General  Act  of  1978,  have  been  given  substantial 
independence  and  powers  to  combat  waste,  fraud,  and  abuse,  including  implied  authority  to  conduct  inspections,  which 
includes  program  evaluation.  IG  inspection  standards  are  spelled  out  in  PCIE/ECIE,  Quality  Standards  for  Inspections, 
at  http://www.ignet.gov.  For  coverage  of  relevant  units  within  IG  offices,  see  PCIE/ECIE,  2007  Survey.  The  IGs  now 
also  belong  to  a  single  collective  council  established  by  law.  The  Inspector  General  Reform  Act  of  2008  (P.L.  1 10-409; 
122  Stat.  4302)  created  a  Council  of  the  Inspectors  General  on  Integrity  and  Efficiency  (CIGIE),  which  merged,  along 
with  other  units,  two  previous  IG  councils:  the  President’s  Council  on  Integrity  and  Efficiency  (PCIE)  and  the 
Executive  Council  on  Integrity  (ECIE).  The  new  Council  has  been  charged  with  using  inspections  and  program 
evaluations,  along  with  audits  and  investigations,  to  handle  problems  which  exceed  the  capacity  or  jurisdiction  of  an 
individual  agency  (122  Stat.  4307).  Reinforcing  its  audit  and  inspection/evaluation  activities,  CIGIE  has  adopted,  as 
one  of  its  three  major  goals,  contributing  to  “government-wide  improvements  in  program  integrity,  efficiency,  and  cost- 
effectiveness  by  providing  cross-agency  analysis  of  OIG  findings  and  recommendations  in  areas  of  vulnerability 
confronting  multiple  government  programs.”  Phyllis  K.  Fong,  Inspector  General,  U.S.  Department  of  Agriculture,  and 
Chair,  CIGIE,  "The  IG  Reform  Act  and  the  New  IG  Council:  Dawn  of  a  New  Era,”  Journal  of  Public  Inquiry, 
Fall/Winter,  2008-2009,  p.  5.  CIGIE  also  includes  a  standing  Committee  on  Inspections  and  Evaluation,  as  had  its 
predecessors  (see  note  8,  above).  The  Committee’s  website,  incidentally,  provides  “related  links”  to  AEA  and  GAO,  at 
http://www.ignet.gov/pande/iande  1  .html. 

20  OMB,  Guide  to  the  Program  Assessment  Rating  Tool  (PART),  Guidance  No.  2008-01,  January  2008,  pp.  1,  30-36, 
and  59-60.  For  a  GAO  study  of  this  part  of  PART,  introduced  in  the  George  W.  Bush  Administration,  see  GAO. 

OMB’s  PART  Reviews.  In  2009,  the  Obama  Administration  developed  its  own  approach:  OMB.  Increased  Emphasis  on 
Program  Evaluation,  Memorandum  for  the  Heads  of  Executive  Departments  and  Agencies,  M-10-01,  October  7,  2009. 
OMB  expanded  upon  this  the  next  year  in  two  memoranda:  OMB,  Evaluating  Programs  for  Efficacy  and  Cost- 
(continued...) 
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Concepts,  Understandings,  and  Specifications  of  "Evaluation"  and 
"Independence" 

These  different  sources  cover  common  ground,  including  the  concepts  of  evaluation  and 
independence,  ways  in  which  the  terms  may  be  understood,  and  how  the  concepts  may  be 
specified.  Differences  exist,  especially  in  how  detailed,  specific,  and  elaborate  these  descriptions 
are,  ranging  from  a  short  statement  on  a  concept  to  a  lengthy  listing  and  interpretation  of  relevant 
standards. 

American  Evaluation  Association 

AEA,  which  promotes  evaluation  for  public  programs,  adopts  a  broad  understanding  of  the 
concept  of  evaluation,  which  can  occur  throughout  the  life  of  a  program:  “Evaluation  is  a  field 
that  applies  systematic  inquiry  to  help  improve  programs,  products,  and  personnel,  as  well  as  the 
human  actions  associated  with  them.”21  While  recognizing  that  evaluators’  work  can  vary  greatly, 
AEA  holds  that 

the  common  ground  for  all  evaluators  is  that  they  aspire  to  achieve  accountability  and 
learning  by  providing  the  best  possible  information  that  might  bear  on  the  value  of  whatever 
is  being  evaluated....  Evaluations  prepared  by  professional,  independent  evaluators  help 
prevent  information  gaps  by:  improving  knowledge  and  understanding  of  how  programs 
work,  strengthening  public  accountability,  assessing  program  effectiveness  and  efficiency, 
and  identifying  opportunities  and  pathways  to  achieving  objectives,  outcomes,  and 
efficiencies."" 

To  support  these  goals,  AEA  specifies  “several  of  the  key  elements  of  a  national  framework  for 
evaluation  practices.”23  These  benchmarks  include 

•  using  appropriate  professional  standards  in  conducting  the  work; 

•  stating  program  goals  and  objectives  as  specifically  as  possible; 

•  issuing  performance  measures  when  the  program  is  being  developed  and  modifying 
them  as  appropriate  to  reflect  what  has  been  learned; 

•  specifying  necessary  requirements  and  resources,  which  “should  be  embedded  in  the 
authorizing  legislation  and  regulations”; 

•  supporting  department-wide  or  government-wide  (i.e.,  GAO)  evaluators  “with  the 
resources,  organizational  independence,  competencies,  and  authorities  necessary  for  the 
effective  evaluation  and  oversight  of  public  programs”; 


(...continued) 

Efficiency,  Memorandum  for  the  Heads  of  Executive  Departments  and  Agencies,  M-10-32,  July  29,  2010;  and 
Performance  Improvement  Guidance:  Management  Responsibilities  and  Government  Performance  and  Results  Act 
Documents,  Memorandum  from  Shelley  Metzenbaum,  OMB  Associate  Director  for  Performance  and  Personnel 
Management,  June  25,  2010. 

21  AEA,  Evaluation,  p.  1. 

22  Ibid. 

23  Ibid.,  p.  4. 
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•  using  private  evaluators  “with  a  broad  range  of  viewpoints  and  capabilities  that  can 
provide  effective  independent  evaluation  as  well  as  input  and  feedback  to  internal 
evaluation  efforts”; 

•  producing  a  wide  range  of  studies,  recognizing  the  advantages  and  limitations  of 
various  methodological  approaches;  and 

•  collaborating  with  stakeholders.24 

In  July  2004,  AEA  approved  a  set  of  guiding  principles  for  evaluators,  which  interrelate  with  one 
another  as  well  as  with  the  foregoing  standards.25  The  five,  each  of  which  is  detailed  in  its 
brochure,  are: 

A.  Systematic  Inquiry:  Evaluators  conduct  systemic,  data-based  inquiries. 

B.  Competence:  Evaluators  provide  competent  performance  to  stakeholders. 

C.  Integrity/Honesty:  Evaluators  display  honesty  and  integrity  in  their  own 
behavior,  and  attempt  to  ensure  the  honesty  and  integrity  of  the  entire  process. 

D.  Respect  for  People:  Evaluators  respect  the  security,  dignity  and  self-worth  of 
respondents,  program  participants,  clients,  and  other  stakeholders. 

E.  Responsibilities  for  General  and  Public  Welfare:  Evaluators  articulate  and  take 
into  account  the  diversity  of  general  and  public  interests  and  values  that  may  be 
related  to  the  evaluation.26 

Encyclopedia  of  Evaluation 

The  Encyclopedia  of  Evaluation  offers  a  working  definition  of  “independent  evaluation”: 

For  an  evaluation  to  be  considered  independent,  the  evaluator  must  be  impartial,  objective, 
unencumbered,  and  balanced.  Further,  because  perceived  independence  is  as  important  as 
independence  itself,  the  evaluator  must  be  accountable  for  every  step  in  the  research  process 
and  able  to  document  all  key  decisions  and  actions  for  the  client  organization,  or  other 
evaluators,  and  the  community  at  large.  Overall,  external  evaluations  tend  to  hold  more 
credibility  than  internal  ones  because  the  external  evaluator  appears  to  have  less  to  gain  or 
lose  from  the  evaluation  findings  and  is  less  likely  to  experience  a  conflict  of  interest."7 

Notably,  this  definition  does  not  appear  to  equate  automatically  an  evaluator’s  “external”  status 
with  independence,  or  an  evaluator’s  “internal”  status  with  lack  of  independence. 


24  Ibid. 

25  AEA,  Guiding  Principles. 

26  Ibid.,  pp.  2-6  (Bold  in  original). 

27  Barrington,  “Independent  Evaluation.” 
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Joint  Committee  on  Standards  for  Educational  Evaluation 

The  Joint  Committee’s  standards  for  evaluating  educational  programs  might  be  adapted  to  other 
fields. 2S  It  posits  that  “sound  evaluations  of  educational  programs,  projects,  and  materials  in  a 
variety  of  settings  should  have  four  basic  attributes.”29  The  associated  standards  govern: 

•  Utility,  which  is  intended  “to  ensure  that  an  evaluation  will  serve  the  information 
needs  of  the  intended  user.”  These  include  stakeholder  identification,  information  scope 
and  selection,  evaluator  credibility,  values  identification,  report  timeliness  and 
dissemination,  report  clarity,  and  evaluation  impact. 

•  Feasibility,  which  is  intended  “to  ensure  that  an  evaluation  will  be  realistic,  prudent, 
diplomatic,  and  frugal.”  Specifics  here  cover  practical  procedures,  political  viability,  and 
cost  effectiveness. 

•  Propriety,  which  is  intended  “to  ensure  that  an  evaluation  will  be  conducted  legally, 
ethically,  and  with  due  regard  for  the  welfare  of  those  involved  in  the  evaluation  as  well 
as  those  affected  by  its  results.”  These  concerns  involve  formal  agreements,  rights  of 
subjects,  complete  and  fair  assessment,  disclosure  of  findings,  dealing  with  conflicts  of 
interest  (actual,  perceived,  potential),  and  fiscal  responsibility. 

•  Accuracy,  which  is  intended  “to  ensure  that  an  evaluation  will  reveal  and  convey 
technically  adequate  information  about  features  that  determine  the  worth  of  merit  of  a 
program.”  The  specifics  here  deal  with:  program  documentation;  context  analysis; 
described  procedures  and  purposes;  defensible  information  sources;  valid,  reliable,  and 
systematic  information;  analysis  of  quantitative  information;  justified  conclusions; 
impartial  reporting;  and  metaevaluation  (that  is,  a  means  of  comparing  a  particular 
evaluation  against  the  standards  developed  for  its  field  along  with  other  pertinent 
standards,  in  order  to  examine  its  strengths  and  weaknesses).30 

Government  Accountability  Office 

GAO  explains  the  concept  of  program  evaluation  by  including  within  its  scope  evaluations 
conducted  by  external  entities,  experts  inside  the  agency  that  contains  a  program,  and  the 
employees  responsible  for  implementing  programs  and  policies: 

Program  evaluations  are  individual  systematic  studies  conducted  periodically  or  on  an  ad  hoc 
basis  to  assess  how  well  a  program  is  working.  They  are  often  conducted  by  experts  external 
to  the  program,  either  inside  or  outside  the  agency,  as  well  as  by  program  managers.  A 
program  evaluation  typically  examines  achievement  of  program  objectives  in  the  context  of 
other  aspects  of  program  performance  or  in  the  context  in  which  it  occurs.  Four  main  types 
can  be  identified,  all  of  which  use  measures  of  program  performance,  along  with  other 
information,  to  learn  the  benefits  of  a  program  or  how  to  improve  it.31 


28  See  Stufflebeam,  "Joint  Committee  for  Education  Evaluation.” 

25  Joint  Committee,  Program  Evaluation  Standards,  pp.  223-224. 

30  Ibid. 

31  GAO.  Performance  Measurement  and  Evaluation,  p.  2.  The  four  types  are  process  (or  implementation)  evaluation, 
outcome  evaluation,  impact  evaluation,  and  cost-benefit  and  cost-effectiveness  analyses.  Ibid.,  p.  4.  See  also  footnote 
18  above. 
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Inspector  General  Community 

The  2008  Inspector  General  Reform  Act  has  directed  the  new  Council  of  the  Inspectors  General 
on  Integrity  and  Efficiency  to  “develop  plans  for  coordinated,  Governmentwide  activities  that 
address  these  problems  and  promote  economy  and  efficiency  in  Federal  programs  and  operations, 
including  interagency  and  interentity  audit,  investigation,  inspection,  and  evaluation  programs 
and  projects  to  deal  efficiently  and  effectively  with  those  problems  concerning  fraud  and  waste 
that  exceed  the  capability  or  jurisdiction  of  an  individual  agency  or  entity.”32  This  call  to  action  to 
all  IGs  builds  on  a  growing  movement  among  individual  offices  to  increase  and  enhance  program 
evaluation. 

The  IG  community  generally  has  considered  evaluation  as  part  of  a  more  encompassing  function 
of  inspection:  “An  inspection  is  defined  as  a  process  that  evaluates,  reviews,  studies,  and/or 
analyzes  the  programs  and  activities  of  a  Department/ Agency”  for  a  number  of  purposes.33  The 
CIGIE  Inspection  and  Evaluation  Committee,  which  drafted  the  standards,  included  among  these 
purposes  providing  factual  information  to  managers  for  decisionmaking;  monitoring  compliance; 
measuring  performance;  assessing  the  efficiency  and  effectiveness  of  programs  and  operations; 
sharing  best  practices;  identifying  where  administrative  action  may  be  necessary;  and  making 
recommendations  for  improvements  to  programs,  policies,  or  procedures.34  To  accomplish  this, 
the  I&E  Committee  developed  standards  dealing  with  14  separate  matters:  competency; 
independence;  professional  judgment;  quality  control;  planning;  data  collection  and  analysis; 
evidence;  records  maintenance;  timeliness;  fraud,  other  illegal  acts,  and  abuses;  reporting;  follow¬ 
up;  performance  measurement;  and  working  relationships  and  communications.35 

The  Inspection  and  Evaluation  Committee,  however,  qualifies  the  adoption  of  the  standards: 
“While  these  standards  are  advisory,  and  compliance  is  voluntary,  their  consistent  application  is 
encouraged.”36  The  panel  adds  that  “the  inspection  function  at  each  Department/ Agency  is 
tailored  to  the  unique  mission  of  the  respective  Department/ Agency.”37  Although  the  inspection 
function  did  not  originate  when  the  IG  Act  was  approved  in  1978,  it  has  grown  over  time  in  terms 
of  complexity  and  diversity  as  well  as  size;  that  is,  the  number  of  OIG  units,  budget,  and  staff 
dedicated  to  it.38  The  result  is  that  most — but  not  all — IG  offices  conduct  inspections;  and  these 
I&E  units  vary  in  expertise,  location,  type  of  work,  longevity,  staffing,  and  budget.39 

Office  of  Management  and  Budget 

Over  the  past  decade,  OMB  has  also  offered  guidance  on  how  to  achieve  independence  and 
expertise  in  evaluations.  Its  directions  and  instructions,  however,  exhibit  some  differences  in 
specifications  and  orientation  between  the  Bush  and  Obama  Administrations. 


32  P.L.  1 10-409.  122  Stat.  4307. 

33  PCIE/ECIE,  Quality  Standards  for  Inspections,  p.  i.  See  footnotes  8,  10,  and  19  above  for  further  discussion  of  the 
IGs’  roles  in  evaluation  and  their  independence. 

34  Ibid. 

35  Ibid,  pp.  1-28. 

36  Ibid.,  cover  letter. 

37  Ibid. 

38  PCIE/ECIE,  2007  Sun’ey,  cover  letter  and  pp.  i  and  1. 

39  Ibid. 


Congressional  Research  Service 


11 


Independent  Evaluators  of  Federal  Programs:  Approaches,  Devices,  and  Examples 


George  W.  Bush  Administration  and  PART 

Considerations  about  the  independence  and  orientation  of  program  evaluation  were  included  in 
the  operation  of  the  Program  Assessment  Rating  Tool  (PART),  which  was  developed  during  the 
Administration  of  George  W.  Bush  but  is  no  longer  in  effect.40  Although  the  guidance  did  not 
define  independent  evaluation  or  evaluator  expressly,  OMB  referred  to  its  role  in  several 
instances.  One  instance,  about  the  scope  and  quality  of  “independent  evaluations,”  stated, 

Purpose:  To  ensure  that  the  program  or  agency  conducts  non-biased  evaluations  on  aregular 
or  as-needed  basis  to  fill  gaps  in  performance  information.  These  evaluations  should  be  of 
sufficient  scope  and  quality  to  improve  planning  with  respect  to  the  effectiveness  of  the 
program.41 

OMB  added  that  such  independent  evaluations  were  to  be  of  “high  quality,  sufficient  scope, 
unbiased,  independent,  and  conducted  on  a  regular  basis  to  support  program  improvements.”42 

As  noted  previously  in  this  report,  however,  GAO  discussed  how  disagreements  arose  in  this 
context  over  how  to  define  independence  for  an  evaluator  and  whether  independence  was 
necessary  or  of  value  in  some  circumstances.43  OMB  initially  advocated  for  a  separate,  outside 
entity — in  particular,  statutory  IGs  or  GAO — to  conduct  the  evaluation.  Agencies  balked  at  this, 
insisting  that  experienced  program  evaluation  offices  within  an  agency,  for  instance,  could 
conduct  such  evaluations  with  independence.  OMB  reconsidered  its  stand  and  allowed  that 
“evaluations  could  be  considered  independent  if  the  program  contracted  them  out  to  a  third  party 
or  they  were  carried  out  by  an  agency’s  program  evaluation  office.”44  OMB  also  recognized  a 
possible  conflict  between  competing  analyses:  “The  program  should  defend  differences  if  an 
independent  entity’s  analysis  differs  from  the  program’s  analysis.”45 

Barack  Obama  Administration 

After  Barack  Obama  became  President,  the  Office  of  Management  and  Budget  issued  several 
memoranda  and  a  guidance  dealing  with  independent  program  evaluation.  OMB  Director  Peter  R. 
Orszag  issued  two  memoranda  on  the  subject — one  in  2009  and  another  the  next  year — which 
began  with  a  nearly  identical  premise:  “Rigorous,  independent  program  evaluations  can  be  a  key 
resource  in  determining  whether  government  programs  are  achieving  their  intended  outcomes  as 
well  [effectively]  as  possible  and  at  the  lowest  possible  cost.”46 

By  comparison  to  the  PART  pronouncements,  the  Obama  OMB  assumed  a  somewhat  different 
stance  on  how  to  define  independence.  In  the  October  2009  memorandum,  OMB  concentrated 


40  OMB,  Guide  to  PART,  pp.30-36  and  59-60.  See  also,  GAO. 

41  OMB,  Guide  to  PART,  pp.  30  and  60. 

42  Ibid. 

43  GAO,  Program  Evaluation:  OMB’s  PART  Reviews  Increased  Agencies  ’  Attention  to  Improving  Evidence  of 
Program  Results,  GAO-06-76,  October  2005. 

44  GAO.  OMB’s  PART  Reviews,  p.  25. 

45  OMB,  Guide  to  PART,  p.  36. 

46  OMB,  Increased  Emphasis  on  Program  Evaluations  (2009),  p.  1 ;  and  Evaluating  Programs  for  Efficacy  and  Cost- 
Efficiency  (2010),  p.  1,  whose  single-word  difference  is  bracketed. 
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more  on  the  concept  of  independent  evaluation  instead  of  an  independent  evaluator.47  Notably, 
this  memorandum  made  repeated  reference  to  conducting  “rigorous,  independent  studies  that  are 
free  from  political  interference.”  The  memorandum  did  not  provide  more  detail  on  what 
constitutes  political  interference  or  what  this  sort  of  independence  might  look  like.  When  an 
agency  applied  for  Administration  support  in  the  President’s  budget  for  capacity-building  funds, 
however,  OMB  required  agencies  to  address  explicitly  their  ability  to  conduct  studies  that  are  free 
from  political  interference. 

In  its  follow-up  memorandum,  OMB  added  to  this  in  its  “evaluation  initiative  for  FY  2012,” 
allocating,  in  the  President’s  upcoming  budget  proposal,  “approximately  $100  million  to  support 
35  rigorous  program  evaluations  and  evaluation  capacity-building  proposals  across  the  Federal 
government.”48  In  so  doing,  the  evaluation  initiative  repeated  the  earlier  call  for  an  “assessment  of 
agency  capacity  to  conduct  rigorous,  independent  evaluations  ...  that  are  free  from  political 
interference.”49  A  complementary  goal — which  also  suggested  several  ways  to  institutionalize 
independent  evaluation  within  an  agency — is  to 

attract  and  retain  talented  researchers  in  an  office  with  standing  within  the  agency.  Agencies 
are  encouraged  to  propose  changes  or  reforms  that  are  needed  to  meet  these  objectives  and 
may  request  funds  to  strengthen  their  internal  evaluation  expertise  and  processes.  Several 
existing  models  may  be  worthy  of  consideration,  including  a  congressionally  chartered 
institute  within  an  agency,  or  an  office  headed  by  a  senior  official  reporting  directly  to  the 
Secretary  or  Deputy  Secretary.59 

The  2009  memorandum  had  also  announced  an  “increased  emphasis”  on  program  evaluations 
and,  more  specifically,  “impact  evaluations.”51  The  2010  memorandum  followed  suit,  recognizing 
the  need  to  overcome  the  absence  of  completed  or  timely  program  evaluations  for  many  federal 
programs.  As  a  result,  the  Office  called  on  agencies  to  include  funding  for  FY2011  to  conduct 
evaluations  or  build  capacity,  to  correct  these  defects  and  strengthen  program  evaluation.52 
Extending  this  orientation,  another  OMB  memorandum  emphasized  a  change  in  the  objective  of 
evaluations  and  their  measurements.  This  OMB  memorandum,  also  issued  in  mid-2010,  states 
that 


the  Administration  is  transitioning  from  a  planning  and  reporting  approach  focused  primarily 
on  the  supply  of  performance  information  to  ....  performance  improvement  strategies  .... 
[with]  Unrelenting  attention  to  achieve  the  ambitious,  near-term  performance  goals  that 
agency  leaders  identify  as  High  Priority  Performance  Goals  (“Priority  Goals”)  in  the 
President’s  FY  2011  Budget;  Establishing  constructive  performance  review  processes  in 


47  OMB,  Increased  Emphasis  on  Program  Evaluations,  Memorandum.  M- 10-01.  October  7,  2009. 

48  OMB,  Evaluating  Programs  for  Efficacy  and  Cost-Efficiency  (2010),  p.  1. 

49  Ibid.,  p.  4. 

50  Ibid. 

51  An  impact  evaluation  seeks  to  quantitatively  estimate  the  extent  to  which  a  program  affects  an  outcome  of  interest 
(e.g.,  the  unemployment  rate)  compared  to  what  would  have  happened  without  the  program.  Ibid.  The  subsequent 
OMB  memorandum  from  the  Director  also  focused  on  “impact  evaluations,  or  evaluations  aimed  at  determining  the 
causal  effects  of  programs  or  particular  strategies,  interventions,  and  within  programs.”  OMB,  Evaluating  Programs  for 
Efficacy  and  Cost-Efficiency  (2010),  p.  2. 

52  OMB,  Increased  Emphasis  on  Program  Evaluations  (2009)  and  Evaluating  Programs  for  Efficacy  and  Cost- 
Efficiency  (2010),  p.  1.  The  goal  is  “that  each  non-security  agency  submit  a  budget  request  5  percent  below  the 
agency’s  FY  2012  discretionary  total  in  the  FY  201 1  Budget.”  OMB.  Evaluating  Programs  for  Efficacy  and  Cost- 
Efficiency,  p.  1,  which  details  the  process  for  addressing  the  problems  and  strengthening  program  evaluation. 
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agencies  that  are  sustained  over  time;  and  Making  Government  Performance  and  Results  Act 
documents  useful.53 


Government  Accountability  Office  Review  in  2010 

A  2010  report  by  the  Government  Accountability  Office,  which  was  released  before  OMB ’s 
initiatives  in  the  same  year,  examined  attempts  by  both  the  Bush  and  Obama  Administrations  to 
improve  government  efficiency,  including  ways  it  connects  to  PART  and  its  successor.54  GAO 
found  that  although  "most  programs  developed  an  efficiency  measure,”  most  of  these  failed  to 
adopt  both  input  and  output  or  outcome  measures;  and  agency  officials  described  “challenges  to 
developing  and  using  program-level  efficiency  measures  and  performance  measures  in  general.”55 
Nonetheless,  “officials  for  some  programs  stated  that  the  efficiency  measures  reported  for  PART 
were  useful”  in  a  variety  of  ways,  including  “to  evaluate  proposals  from  field  units.”56  The  GAO 
report  also  recognized  improvements  coming  from  both  the  Bush  and  Obama  Administrations 
along  with  the  lengthy  history  of  legislative  reform  efforts  that  tie  efficiency  into  program 
effectiveness,  particularly  in  the  Government  Performance  and  Results  Act.57 

The  two  subsequent  2010  OMB  memoranda  recognized  some  of  the  same  goals.  Its  memorandum 
on  Evaluating  Programs  for  Efficacy  and  Cost-Efficiency  (2010),  as  the  title  suggests,  dealt  with 
the  relationship  between  efficacy  and  efficiency  and  the  need  to  measure  both.  And  OMB’s 
Performance  Improvement  Guidance  added  to  this,  specifying  that  evaluation  should  focus  on 
“performance  improvement  strategies”  in  general  and  on  “making  GPRA  documents  useful”  in 
particular.58 


Possible  Criteria  and  Attributes  for  an  Independent  Evaluation 
Unit  or  Function 

The  aforementioned  understandings  of  terms  like  “evaluation”  and  “independence”  are  not  hard- 
and-fast  rules.  Rather,  they  might  be  considered  as  perspectives,  some  of  which  are  similar.  Using 
the  perspectives  and  some  of  the  examples  cited  later  in  this  report,  a  number  of  possible  criteria 
for  an  independent  evaluator  can  be  identified.  These  characteristics,  which  are  not  exhaustive, 
are  intended  only  to  identify  a  range  of  possibilities  and  options  to  consider  for  establishing  an 
independent  evaluator,  modifying  evaluative  operations  and  organizations,  or  reviewing  an 
evaluator’s  methodology  and  findings.  CRS  takes  no  position  on  the  advisability  of  independence 
or  the  adoption  of  any  particular  characteristic. 


53  OMB,  Performance  Improvement  Guidance,  pp.  1-2. 

54  GAO.  Streamlining  Government:  Opportunities  Exist  to  Strengthen  OMB’s  Approach  to  Improving  Efficiency, 
Report  GAO-10-394.  The  major  portion  of  this  GAO  performance  audit  was  conducted  from  September  2008  to  May 
2010.  Ibid.,  p.  5. 

55  Ibid.,  Summary. 

56  Ibid. 

57  Ibid.,  Summary  and  pp.  2-4. 

58  OMB.  Performance  Improvement  Guidance,  pp.  1-2. 
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Possible  Criteria  Addressing  the  Independence  of  an  Evaluator 

Based  on  the  foregoing  analysis  and  the  examples  cited  later  in  this  report,  the  following  criteria 
illustrate  a  range  of  factors  that  could  affect  the  independence  of  an  IE-like  entity,  which,  in  turn, 
could  help  to  determine  the  IE’s  capacity,  capability,  credibility,  impact,  and  effectiveness. 

•  Well-defined  jurisdiction.  Specifying  an  independent  evaluator’s  jurisdiction — over 
the  subject  matter,  specific  projects  or  programs,  and  the  agencies  to  be  covered — is 
especially  important  with  regard  to  interagency,  intergovernmental,  and  interstate 
programs. 

•  Clear  separation  from  the  program  or  project  office.  A  range  of  options  exist 
here.  For  instance,  this  might  occur  through  an  evaluation  unit  which  is  external  to  the 
office,  its  parent  agency,  or  both.  An  outside  unit  could  be  established  in  a  number  of 
ways:  setting  up  a  new  office,  operating  under  standards  and  guidelines  that  protect  its 
independence  (e.g.,  prohibiting  revision  by  political  appointees  or  OMB);  contracting 
with  a  private  organization;  making  arrangements  with  a  relevant  government-chartered 
organization;59  or  relying  on  an  appropriate  federal  agency,  such  as  an  IG,  an  executive 
organization  with  relevant  expertise  and  experience,60  or  GAO.  Another  possibility 
involves  an  office  within  the  agency.  The  independence  of  such  an  evaluation  might  also 
be  protected  if  it  were  assigned  to  an  in-house  program  evaluation  office  unconnected 
with  the  program  or  project  implementer. 

•  Avoidance  of  a  (potential)  conflict  of  interest,  actual  or  perceived.  For  an 

evaluator,  this  might  mean,  among  other  considerations,  no  direct  ties  to  the  program  or 
project  office,  its  implementing  agents,  or  affected  parties. 

•  Neutrality,  objectivity,  and  impartiality  in  conducting  an  evaluation  as  well  as  in 
reporting  and  disseminating  its  findings,  conclusions,  and  recommendations.  Even  if 
an  evaluation  entity  were  not  housed  separately  from  the  program  being  implemented, 
third-party  review  of  the  evaluation,  for  example,  might  detect  or  deter  bias,  providing 
some  assurance  of  independence.  In  addition,  evaluations  might  be  required  to  focus  on 
several  competing  definitions  of  programmatic  “success,”  if  the  proper  goals  of  a 
government  activity  were  contested. 

•  Appropriate  criteria  and  standards  for  conducting  an  evaluation  and  for  basing 
its  conclusions.  Such  criteria  and  standards  would  be  determined  by  the  subject  matter 
being  assessed  as  well  as  by  the  extent  and  detail  of  the  coverage  called  for.  This  attribute 
for  an  independent  evaluation  would  serve  to  justify  the  validity  and  reliability  of  its 
findings  and  conclusions. 


59  Several  national  academies — National  Academy  of  Sciences,  National  Academy  of  Engineering,  Institute  of 
Medicine,  and  National  Research  Council — have  been  called  upon  to  provide  independent  evaluations  and  assessments 
directly  or  to  advise  agency  officials  on  these.  For  a  listing  of  these  charges  from  the  106lh  to  the  1 11th  Congress,  see 
U.S.  National  Academies,  Short  Summaries  of  National  Academies’  Studies  Within  Public  Laws  by  Congressional 
Session,  available  at  http://www7.nationalacadernies.org/ocga/Other/Short_Summaries. 

60  An  illustration  of  this  involves  the  National  Institute  of  Science  and  Technology  (NIST)  and  its  review  of  the 
Department  of  Agriculture’s  (USDA)  National  Organic  Program  (NOP).  According  to  USDA,  the  NOP  has  applied  to 
NIST  “for  recognition  of  its  accreditation  function....  Applying  for  and  receiving  recognition  will  support  NOP’s 
credentials  as  an  accrediting  body  and  satisfy  regulatory  requirements  to  obtain  peer  review....  Third-party  recognition 
is  important  for  many  of  USDA’ s  audit-based  programs.’’  Kathleen  A.  Merrigan,  Deputy  Secretary,  USDA,  letter,  June 
4,  2009,  to  Steve  Etka,  Executive  Director,  National  Organic  Coalition,  Washington,  DC. 
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•  Specified  type(s)  of  evaluation.  GAO  identifies  these  as  cost-effectiveness, 
implementation,  impact,  and  outcome.  Refinements  of  these  and  other  possibilities  might 
be  considered  as  well. 

•  Designated  responsibilities  and  duties.  These  could  extend  beyond  assessing 
program  performance,  to  developing  appropriate  measures,  providing  information  and 
data  to  operational  offices,  monitoring  compliance  with  program  directives,  issuing 
recommendations  for  corrective  action,  sharing  best-practices,  and  responding  to  and 
reconciling  concerns  or  criticisms  from  report  subjects. 

•  Competency  and  expertise  (in  the  subject  area,  in  evaluation,  or  in  both).  This 
applies  to  both  the  principal  evaluator  and  staff. 

•  Sufficient  resources.  Resources  for  the  IE  or  similar  posts  include  funding,  staff, 
and,  if  necessary,  contracting  authority,  appropriate  equipment  and  facilities,  among  other 
items. 

•  Access  to  and  use  of  relevant  information  and  data  which  are  reliable  and  valid. 

Program  evaluations — and  their  credibility — rely  upon  the  reliability  and  validity  of  the 
data  and  information  that  they  use  and  that  are  appropriate  for  the  subject. 

•  Accounting  for  and  documenting  steps  in  the  evaluation  process.  These  extend  to 
the  measurements  used  as  well  as  the  selection  of  information  and  data  relied  upon,  to 
defend  decisions  if  necessary. 

•  Report  contents  and  timetables.  These  include  what  the  reports  should  contain 
(findings,  conclusions,  and/or  recommendations)  and  when  they  should  be  submitted  (on 
a  fixed  time  schedule,  after  certain  conditions  are  met,  at  different  stages  of  an  evaluation, 
or  at  the  end  of  the  project  or  program). 

•  Report  recipients,  availability,  and  dissemination.  This  involves  who  is  to  receive 
the  IE  reports  (Congress,  the  President,  agency  officials,  or  the  general  public)  and  how 
an  IE  report  is  to  be  made  available  (in  what  medium  and  at  what  cost,  if  any). 

•  Evaluation  responses  and  report  revisions.  This  specifies  whether  a  report  may  be 
revised  by  the  head  of  an  agency,  OMB,  or  the  President,  or  whether  responses  from  the 
evaluation’s  subjects  are  required.  Responses  to  a  report’s  findings,  conclusions,  and 
recommendations  are  particularly  important  from  the  program  or  project  office. 

•  Means  to  resolve  differences  between  an  evaluator  and  the  operational  office. 

Various  means  might  be  considered  to  resolve  or,  alternatively,  transparently  report,  any 
differences  in  the  event  that  the  program  office  disagrees  with  an  evaluator’s  findings, 
conclusions,  and  recommendations. 

Possible  Attributes  When  Designing  or  Reviewing  an  Independent  Evaluation 
Unit 

These  criteria  also  suggest  a  range  of  considerations  in  establishing  a  new  IE  entity,  or  reviewing 
and  modifying  an  existing  one.  A  number  of  choices  about  organizational  design  and  operations 
might  be  considered  that  could  support  the  independence,  quality,  reliability,  and  meaningfulness 
of  an  IE’s  studies  and  reports  to  a  variety  of  stakeholders,  including  Congress,  the  President, 
agency  employees,  state  and  local  governments,  interest  groups,  private  contractors,  and  the 
public.  These  outcomes  are  likely  to  be  enhanced  by  clear  and  specific  requirements  regarding  the 
purposes,  responsibilities,  and  duties  of  the  IE  entities  as  well  as  protections  for  their  officers. 
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Posed  as  questions,  these  considerations,  which  closely  relate  to  one  another  and  in  some 
circumstances  may  overlap,  include  the  following: 

•  Selection  Requirements:  Are  any  requirements  or  criteria  for  an  IE’s  selection 
specified  in  the  authority  creating  the  entity  or  position?  If  so,  is  he  or  she  to  be  selected 
for  relevant  expertise  or  experience  in  the  subject  area  or  in  evaluation? 

•  Appointing  Authority  and  Requirements:  Who  appoints  the  individual  who  will 
lead  the  IE  entity?  If  the  independent  evaluator  is  an  individual,  who  will  appoint  the 
individual?  Is  the  appointment  made  by  a  certain  agency  official  (head  of  the  agency  or  of 
a  program  office)?  Is  it  an  individual  officer  or  a  committee  (in  the  case  of  an  intra¬ 
agency,  interagency,  or  intergovernmental  unit)?  If  the  appointment  is  made  by  the  chair 
of  a  committee,  is  he  or  she  directed  to  consult  with  other  panel  members?  Is  the 
appointing  authority  advised  to  or  required  to  consult  with  inspectors  general  or  with 
outside  organizations  in  determining  prospective  candidates? 

•  Tenure:  How  long  is  an  evaluation  officer’s  tenure?  Is  it  confined  to  a  short-term 
project  or  a  fixed  deadline,  such  as  six  months  or  one  year?  Or  does  it  extend  indefinitely, 
throughout  the  life  of  an  entire  program? 

•  Funding:  How  is  the  entity  funded?  Are  the  funds  from  an  administrative  account  at 
the  discretion  of  an  agency  or  program  official?  Or  is  the  funding  a  specified  amount  or 
percentage  of  the  overall  budget  of  a  program  or  project? 

•  Supervision:  Who,  if  anyone,  supervises  the  IE’s  operations? 

•  Purposes:  Why  exactly  has  an  office  been  established,  and  what  is  it  expected  to 
accomplish?  Is  it  to  conduct  an  evaluation  over  a  long-term  broad  program,  a  short-term 
narrow  project,  or  a  series  of  these?  Is  the  unit  to  offer  recommendations,  as  well  as 
findings  and  conclusions?  What  roles  is  an  evaluation  unit  expected  to  play  in  the 
ongoing  operation  of  the  program  or  project,  in  advising  an  overseer  or  program  manager, 
or  in  the  final  assessment  of  the  program?  Are  the  IE’s  findings,  conclusions,  and 
recommendations  expected  to  contribute  to  management  and  best-practices  approaches, 
for  instance,  or  to  examine  the  effectiveness  of  the  program  itself,  separate  from  its 
implementation? 

•  Substantive  Considerations  and  Type  of  Evaluation:  What  substantive  research 
questions  is  it  to  address?  What  type  or  types  of  evaluation  is  the  entity  to  perform?  How 
much  discretion  will  be  necessary  to  leave  to  the  IE  to  make  these  decisions? 

•  Mandated  Studies:  What  particular  studies,  if  any,  are  mandated?  Is  an  IE  required 
to  conduct  certain  studies  (e.g.,  interims  as  well  as  a  final)  or  does  he  or  she  have  some 
discretion  over  the  range  and  frequency  of  studies? 

•  Study  Standards  and  Procedures:  What  standards,  procedures,  and  guidelines,  if 
any,  is  an  IE  expected  to  follow  in  conducting  evaluations?  Are  these  detailed  in  the  unit’s 
establishing  authority? 

•  Consultation  with  Outside  Organizations:  Is  an  IE  entity  required  to  consult  with 
other  governmental  or  nongovernmental  organizations  in  carrying  out  its  responsibilities? 
In  general,  should  the  IE  consult  with  private  sector  specialists  in  evaluation  or  with 
inspectors  general;  in  scientific  matters,  with  one  of  the  relevant  National  Academies  or 
private  organizations;  and  in  public  administration  matters,  as  an  example,  with  the 
National  Academy  of  Public  Administration? 
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•  Types  of  Reports:  What  types  of  reports  are  ordered?  Are  the  reports  limited  to 
particular  projects  or  stages  thereof;  are  their  contents  outlined;  and  are  the  reports  to 
contain  recommendations  for  change? 

•  Reporting  Schedules:  When  are  the  reports  to  be  issued? — according  to  a  fixed  time 
schedule? — at  the  completion  of  a  particular  stage  of  a  project  or  program? — at  the  end  of 
the  entire  program? — or  at  the  discretion  of  the  IE? 

•  Report  Recipients:  Who  is  to  receive  the  reports?  Is  it:  the  operational  office? — the 
agency? — the  President? — Congress? — another  government  entity,  such  as  GAO  or  an 
inspector  general? — the  general  public? 

•  Report  Availability  and  Dissemination:  How  is  the  report  to  be  publicized — by 
way  of  the  IE’s  office,  the  affected  agency  or  program  office,  a  congressional  panel  which 
is  a  recipient,  or  the  news  media?  How  is  it  to  be  disseminated?  Is  it  in  hard-copy  (from 
the  IE  or  the  Government  Printing  Office,  for  instance)  or  on  the  website  of  the  IE,  the 
affected  agency,  or  another  official  recipient? 

•  Response  Obligations:  What  obligations,  if  any,  does  an  affected  agency  or  program 
office  have  to  respond  to  the  IE’s  findings,  conclusions,  and  possible  recommendations? 

•  Considering  or  Resolving  Differences:  Is  there  a  means  to  consider  and  possibly 
resolve  differences  on  such  matters,  if  disagreements  arise  between  the  IE  and  the 
program  or  project  implementer? 

•  Jurisdictions:  What  is  an  IE’s  jurisdiction?  Is  it  over  a  narrow,  short-term  project;  a 
broad,  long-range  program;  or  some  combination  between  these  ends  of  the  spectrum? 

Does  the  jurisdiction  cover  a  single  federal  office  or  agency,  a  number  of  federal 
agencies,  or  a  collection  of  federal,  state,  and  local  government  entities?  Does  the 
jurisdiction  cover  a  domain  of  public  policy? 

•  Modes  of  Operation:  What  is  the  mode  of  operation  of  an  independent  evaluation 
unit?  Is  it  in  continuous  session,  monitoring  and  assessing  a  particular  program  on  an  on¬ 
going  basis?  Or  does  it  operate  sporadically,  evaluating,  for  instance,  several  distinct 
projects  or  a  program  when  it  reaches  certain  stages?  Or  is  it  only  active  when  certain 
conditions  or  a  time  schedule  calls  for  its  involvement? 

Concluding  Observations 

In  the  understanding  adopted  here,  truly  independent  evaluators  are  not  found  within  the  program 
office  itself,  because  that  location  would  (appear  to)  compromise  their  independence.  Even  with 
this  qualification,  however,  independent  evaluators  of  federal  programs  appear  to  have  increased 
since  the  1960s  and  continue  to  do  so.  IEs  exist  both  inside  the  agency  proper — in  auditing 
offices,  for  instance,  or,  as  a  “model”  raised  by  OMB,  in  a  “congressionally  chartered  institute” — 
as  well  as  outside  it.  Independent  evaluators  and  comparable  entities  have  been  created  as  new, 
separate  units,  operating  under  a  variety  of  names  and  titles.  And  evaluations  have  been 
undertaken  by  appropriate  existing  agencies,  including  the  Government  Accountability  Office, 
offices  of  inspector  general,  or  various  other  government  constructs,  such  as  NIST  and  the 
National  Academy  of  Sciences. 
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IEs  and  similar  constructs,  moreover,  are  multi-faceted  and  diverse,  differing  in  their  establishing 
authority,  structure,  organization,  responsibilities,  and  requirements,  among  other  characteristics. 
Such  differences  have  come  about  for  a  number  of  reasons,  including  the  following: 

•  No  agreed-upon,  precise,  and  detailed  definition  of  what  constitutes  “independence” 
and  “evaluation”  exists. 

•  Suggested  or  recommended  standards  and  criteria,  although  having  commonalities, 
are  not  duplicative  in  all  respects,  with  some  being  more  detailed  and  expansive  than 
others.  Nonetheless,  a  number  of  criteria  and  attributes  could  be  considered  in  creating  or 
modifying  an  independent  evaluator. 

•  A  perceived  need  for  flexibility  in  IE  structure,  organization,  operations,  and 
activities  appears  in  some  establishments,  which  allows  the  IE  (or  other  officials)  to 
determine  the  entity’s  features.  By  comparison,  others  are  given  detailed  duties  and 
directions  upon  their  creation. 

•  Demands  and  expectations  differ  among  IEs. 

•  The  political  context  in  which  IEs  are  established  differs  over  time  and  within  and 
among  policy  arenas. 

Because  of  these  and  other  factors,  independent  evaluators  follow  no  single  path  or 
set  of  directions.  Instead,  they  reveal  numerous  ways  and  directives  for  possible 
approaches  to  assess  federal  programs;  provide  relevant  information  and  data  to  the 
executive,  legislature,  stakeholders,  and  the  general  public;  enhance  oversight  of 
affected  programs;  and  aid  in  the  development  of  new  legislation  or  executive 
directives. 
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Appendix.  Examples  of  Independent  Evaluators  and 
Similar  Positions 

Following  are  descriptions  of  a  number  of  independent  evaluation  entities,  as  set  up  by  their 
establishing  authority  or  proposed.  These  were  selected  through  a  search  of  public  laws, 
executive  orders,  other  administrative  directives,  and  legislative  proposals  that  specifically  call 
for  an  “independent  evaluation,”  “independent  evaluator,”  or  similar  devices,  including  “peer 
review”  or  “independent  study.”  The  listing  is  not  comprehensive.  For  instance,  it  does  not 
include  relevant  reports  from  the  Government  Accountability  Office,  because  of  their  substantial 
number.  And  it  does  not  include  relevant  reports  from  inspectors  general,  unless  these  are 
specifically  called  for  in  legislation.  Nonetheless,  the  descriptions  of  the  selections  illustrate  the 
diversity  of  independent  evaluation  units  and  their  adoption  over  time,  particularly  in  the 
contemporary  era.  (For  each  entry,  the  title  of  the  entity  uses  the  lower  or  upper  case  as  it  appears 
in  the  authorization  or  proposal  and  is  in  bold  type  for  emphasis.) 

Based  on  these  examples,  no  single  entity  meets  all  the  characteristics  that  could  be  addressed  in 
its  establishing  authority,  although  some  come  close.  The  selections  also  demonstrate  a  wide 
range  and  diversity  among  entities  which  have  been  empowered  to  conduct  evaluations.  As  a 
corollary,  none  of  the  evaluation  offices  are  identical  to  any  other  in  all  respects.  Differences  are 
observable  in  the  specifics  associated  with:  appointment,  tenure,  jurisdiction,  evaluation  criteria 
and  standards,  consultation  with  outside  organizations,  reporting  mandates,  reports’  recipients  and 
responses,  and  funding  and  staffing,  among  other  characteristics. 


Chesapeake  Bay  Protection  and  Restoration  Independent  Evaluator 
(Executive  Order  13508) 61 

The  independent  evaluator  called  for  in  Executive  Order  13508,  “Chesapeake  Bay  Protection 
and  Restoration,”  issued  by  President  Obama  on  May  12,  2009,  is  to  assist  a  new  Federal 
Leadership  Committee  for  the  Chesapeake  Bay  in  its  oversight  of  the  program.  The  committee, 
established  in  the  order,  comprises  representatives  from  a  number  of  federal  departments  and 
agencies;  it  was  established  “to  begin  a  new  era  of  shared  Federal  leadership  with  respect  to  the 
protection  and  restoration  of  the  Chesapeake  Bay.”62  The  committee’s  responsibilities  are  “to 
oversee  the  development  and  coordination  of  programs  and  activities,”  along  with  managing  “the 
development  of  strategies  and  program  plans  for  the  watershed  and  ecosystem  of  the  Chesapeake 
Bay  and  oversee  their  implementation.”63  Among  other  things,  the  committee  is  to  prepare  and 

61  President  Barack  Obama,  Executive  Order  13508,  Chesapeake  Bay  Protection  and  Restoration .  May  12,  2009,  74 
FR  23099-23104,  May  15,  2009. 

62  EO  13508,  Sec.  201.  The  federal  departments  and  agencies  represented  are:  the  Environmental  Protection  Agency 
(EPA),  whose  Administrator  chairs  the  Committee;  the  Departments  of  Agriculture,  Commerce.  Defense,  Homeland 
Security,  Interior,  and  Transportation;  and  such  other  agencies  as  determined  by  the  Committee.  This  new  development 
ties  into  long-term  efforts  of  the  Chesapeake  Bay  Program,  a  regional  partnership  that  has  existed  since  1983.  The 
Program  consists  of  EPA,  which  represents  the  Federal  government  and  manages  the  Program  Office;  six  states 
(Delaware,  Maryland,  New  York,  Pennsylvania,  Virginia,  and  West  Virginia);  the  District  of  Columbia;  the 
Chesapeake  Bay  Commission,  a  tri-state  legislative  body;  and  participating  citizen  advisory  groups.  Descriptions  of 
actions  taken  by  EPA.  include  EPA,  Guidance  for  Federal  Land  Management  in  the  Chesapeake  Bay,  May  12,  2010, 
and  The  Chesapeake  Bay  Program,  available  http://www/epa.gov/region03/chesapeake/indx.htm. 

63  E.O.  13508,  Sec  201. 
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publish  a  strategy,  following  submission  of  relevant  draft  reports  from  lead  federal  agencies.  The 
overarching  strategy  is  to  include,  among  other  things:  defining  environmental  goals;  identifying 
key  measurable  indicators  or  environmental  condition  and  changes;  describing  specific  programs 
and  strategies  to  be  implemented;  identifying  the  mechanism  to  assure  that  governmental  and 
other  activities,  including  data  collection  and  distribution,  are  coordinated;  and  “describing  a 
process  for  the  implementation  of  adaptive  management  principles,  including  a  periodic 
evaluation  of  protection  and  restoration  activities.”64 

These  requirements,  directly  or  indirectly,  tie  into  the  functions  of  an  independent  evaluator 
called  for  in  the  executive  order.  In  order  to  “strengthen  accountability”  over  the  program, 

[t]he  Committee,  in  collaboration  with  State  agencies,  shall  ensure  that  an  independent 
evaluator  periodically  reports  to  the  Committee  on  progress  toward  meeting  the  goals  of  this 
order.  The  Committee  shall  ensure  that  all  program  evaluation  reports,  including  data  on 
practice  or  system  implementation  and  maintenance  funded  through  agency  programs,  as 
appropriate,  are  made  available  to  the  public  by  posting  on  a  website  maintained  by  the  Chair 
of  the  Committee.65 

Key  characteristics  of  the  Chesapeake  Bay  Restoration  independent  evaluator  position — based  on 
this  provision  and  portions  of  the  Federal  Leadership  Committee’s  mandate — are: 

•  an  interagency  and  inter-governmental  jurisdiction,  commensurate  with  the 
committee’s  jurisdiction; 

•  an  indefinite  tenure  based  on  the  life  of  the  committee; 

•  reporting  requirements  that  authorize  the  IE  to  report  “periodically  ...  on  progress  in 
meeting  the  goals  of  this  order”  to  the  committee;  and 

•  a  directive  to  the  committee  to  make  these  reports  and  relevant  data  available  to  the 
public. 


64  Ibid.,  Sec.  203.  As  described  by  EPA,  “biennial  monitoring,  tracking  and  reporting  will  be  conducted  to  assess  the 
effectiveness  of  implementation  actions  identified  in  the  two-year  milestones  [goals  and  commitments  by  states  to 
reduce  pollution  in  the  Bay],  These  data  will  also  be  used  to  drive  adaptive  decision-making  and  redirect  management 
actions.’’  EPA,  Chesapeake  Bay  Program ,  “Frequently  Asked  Questions,”  p.  7. 

65  Ibid.,  Sec.  206.  Earlier  in  the  executive  order,  the  District  of  Columbia  is  identified  separately  from  Chesapeake  Bay 
area  States.  Later,  the  Committee  is  directed  “to  consult  extensively  with  the  States  of  Virginia,  Maryland, 
Pennsylvania,  West  Virginia,  and  Delaware,  and  the  District  of  Columbia."  Ibid.  Sec.  204.  Because  the  District  of 
Columbia  is  not  a  state,  it  does  not  technically  fall  under  the  rubric  in  Section  206,  which  calls  for  “collaboration  with 
State  agencies.”  By  comparison,  H.R.  1053  Sec.  4(3),  explicitly  includes  the  District  of  Columbia  under  its  definition 
of  “States.” 
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Proposed  Chesapeake  Clean  Water  and  Ecosystem  Restoration  Act 
of  2009  Inspector  General  and  Chesapeake  Bay  Program  Scientific 
and  Technical  Advisory  Committee66 

This  bill  (S.  1816,  111th  Congress),  which  focuses  on  restoration  of  Chesapeake  Bay,  provides  for 
the  Inspector  General  of  the  Environmental  Protection  Agency  to  evaluate  implementation  of 
the  enactment  and  for  a  special  Scientific  and  Technical  Advisory  Committee.  The  legislation  is 
intended  to  expedite  and  enhance  the  restoration  of  the  Chesapeake  water  and  ecosystem.  It 
would  reauthorize  the  Chesapeake  Bay  Program,  expand  state  and  local  government  authority, 
provide  new  grant  authorizations,  and  strengthen  enforcement  tools.  In  so  doing,  it  would 
augment  existing  legislation  (the  Federal  Water  Pollution  Control  Act);  President  Obama’s 
Executive  Order  13508  (described  above);  and  the  on-going  Chesapeake  Bay  Program,  which  has 
been  a  federal,  state,  and  local  government  charge  since  the  mid-1980s.  The  Chesapeake 
Executive  Council  would  direct  the  Program,  in  accordance  with  a  Chesapeake  Bay  Agreement. 
The  Council  would  develop  and  implement  management  strategics  and  plans  for  the  Program. 

The  operational  Office  of  the  Council  would  remain  in  EPA,  providing  support  for  it  in  a  number 
of  ways.  These  include  implementing  and  coordinating  science,  research,  monitoring,  and  data 
collection;  developing  and  maintaining  pertinent  information;  assisting  the  signatories  of  the 
Agreement  in  developing  and  implementing  action  plans;  coordinating  the  activities  of  EPA  and 
other  entities  in  developing  strategics  to  improve  water  quality  and  living  resources  of  the  Bay; 
and  implementing  outreach  programs  for  public  information,  education,  and  participation  to 
foster  stewardship  of  the  bay.  Along  with  this,  the  EPA  Administrator  would  be  authorized  to 
enter  into  agreements  with  other  federal  agencies  to  carry  out  these  matters.  If  enacted,  S.  1816 
would  provide  for  various  grants  to  public  and  private  entities,  for  technical  and  other  assistance 
as  well  as  implementation  and  monitoring. 

In  accordance  with  E.O.  13508,  S.  1816  would  also  require  the  EPA  Administrator  to  issue  an 
annual  action  plan — describing  how  federal  funding  in  the  President’s  budget  submission  to 
Congress  would  be  used  to  protect  and  restore  the  Chesapeake  Bay — and  biennial  progress 
reports  assessing  relevant  developments  and  indicators. 

The  bill  also  provides  that  the  “Inspector  General  of  the  Environmental  Protection  Agency 

shall  evaluate  the  implementation  of  this  section  on  a  periodic  basis  of  not  less  than  once  every 
three  years.”  No  other  particulars  about  such  an  evaluation  are  specified  in  the  proposal.  The 
legislation  would  also  require  the  EPA  Administrator  to  consult  with  several  different  entities 
about  particular  matters.  One  of  these  directives  is  “to  consult  with  the  Chesapeake  Bay 
Program  Scientific  and  Technical  Advisory  Committee  regarding  independent  review  of 
[certain]  monitoring  designs.” 


66  S.  1816,  111th  Cong.,  1st  sess.,  introduced  by  Senator  Benjamin  Cardin  for  himself  and  others.  The  bill  and 
supportive  statements  are  at  Congressional  Record,  vol.  155,  October  20,  2009,  pp,  S10573-S10581.  For  additional 
comments,  see  “Cardin,  O’Malley.  Cummings,  Van  Hollen  and  regional  partners  detail  new  legislation  to  restore  the 
Chesapeake  Watershed,”  press  release,  October  20,  2009;  and  letter  of  support  to  Honorable  Benjamin  L.  Cardin  from 
Maryland  Governor  Martin  O’Malley,  Virginia  Governor  Timothy  M.  Kaine,  and  District  of  Columbia  Mayor  Adrian 
M.  Fenty,  October  20,  2009,  both  available  at  http://www.cardin.senate.gov/news/indx/cm. 
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Proposed  Chesapeake  Bay  Accountability  and  Recovery  Act  of  2009 
Independent  Evaluator67 

This  proposed  legislation  (H.R.  1053,  111th  Congress)  parallels  some  of  the  main  provisions  in 
E.O.  13508,  both  of  which  deal  with  the  Chesapeake  Bay  restoration  and  include  an  Independent 
Evaluator.  The  House  bill,  as  approved  by  the  House  Committee  on  Natural  Resources,  calls  for 
an  “interagency  crosscut  budget”  for  restoration  activities  in  the  Chesapeake  Bay  watershed,  with 
funding,  expenditures,  and  accounting  requirements  detailed  (Sec.  I).68  This  crosscut  budget, 
composed  by  the  various  agencies  involved  in  the  restoration  program,  is  to  be  submitted  to 
Congress  by  the  Director  of  the  Office  of  Management  and  Budget,  after  consultation  with  the 
Chesapeake  Bay  Executive  Council,  the  chief  executive  of  each  Chesapeake  Bay  State  (the 
definition  of  which  includes  the  District  of  Columbia),  and  the  Chesapeake  Bay  Commission.  In 
addition,  H.R.  1053  (Sec.  3)  requires  that  the  Administrator  of  the  Environmental  Protection 
Agency  (EPA),  “in  consultation  with  other  Federal  and  State  agencies,  shall  develop  an  adaptive 
management  plan  for  the  Chesapeake  Bay  Program  and  restoration  activities  that  includes,” 
among  other  matters:  a  definition  of  specific  and  measurable  objectives  to  improve  water  quality, 
habitat,  and  fisheries;  a  process  for  stakeholder  participation;  monitoring,  modeling, 
experimentation,  and  other  research  and  evaluation  practices;  a  process  for  modifying  restoration 
priorities  that  have  not  attained  or  will  not  attain  the  aforementioned  specific  and  measureable 
objectives;  and  a  process  for  prioritizing  restoration  activities  and  programs  to  which  adaptive 
management  is  to  be  applied.  The  EPA  Administrator  is  also  required  to  submit  annual  reports  on 
the  implementation  of  the  adaptive  management  plan. 

The  proposal  adds  an  Independent  Evaluator,  appointed  by  the  Administrator  from  among 
nominees  submitted  by  the  Chesapeake  Executive  Council.  The  IE  is  to  “review  and  report  on 
restoration  activities  and  the  uses  of  adaptive  management,  including  on  such  related  topics  as  are 
suggested”  by  the  Council  (Sec.  4).  The  IE  “shall  submit  a  report  to  the  Congress  every  3  years  in 
the  findings  and  recommendations  of  reviews”  with  regard  to  the  adaptive  management  plans  and 
their  implementation  (Sec.  4).  As  noted  above,  the  plans’  inclusions  are:  developing  measurable 
objectives;  monitoring,  modeling,  experimentation,  and  other  research  and  evaluation  practices; 
and  processes  for  modifying  restoration  activities  as  well  as  prioritizing  restoration  activities  and 
programs. 


67  H.R.  1053,  1 1 1th  Cong.,  1st  sess.,  an  Amendment  in  the  Nature  of  a  Substitute  to  H.R.  1053,  offered  by 
Representative  Wittman.  as  agreed  to  by  the  House  Committee  on  Natural  Resources,  July  29,  2009.  For  a  press 
account,  see  "CQ  Committee  Coverage,”  Congressional  Quarterly ,  July  29,  2009,  p.  2,  at  http://www.cq.com/find.do? 
dataSource=committeemarkups&queryFragment. 

68  One  CRS  study  describes  a  crosscut  budget  as  follows:  “At  its  most  basic  level,  a  crosscut  budget  is  often  used  to 
present  budget  information  from  two  or  more  agencies  whose  activities  are  targeted  at  a  common  policy  goal  or  related 
policy  goals  ....  In  the  context  of  ecosystem  restoration,  a  crosscut  budget  is  typically  a  document  that  organizes  and 
reports  the  activities  and  funding  of  several  entities  within  the  same  broad  initiative  in  a  way  that  ‘cuts  across’ 
organizational  boundaries.”  CRS  Report  RL34329,  Crosscut  Budgets  in  Ecosystem  Restoration  Initiatives:  Examples 
and  Issues  for  Congress ,  by  Pervaze  A.  Sheikh  and  Clinton  T.  Brass.  Analyzing  crosscut  budgeting  in  a  broad  context, 
including  its  possible  policy-making  and  power-broker  effects.  Allen  Schick  emphasized  its  role  as  an  action-forcing 
mechanism,  designed  to  compel  agencies  to  cooperate  in  furtherance  of  a  shared  policy  responsibility.  Allen  Schick, 
"The  Coordination  Option,”  in  Peter  Szanton,  ed..  Federal  Reorganization:  What  Have  We  Learned?  (Chatham  House 
Publishers,  Inc.,  Chatham.  New  Jersey,  1981),  pp.  99-101. 
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Comprehensive  Everglades  Restoration  Plan  Independent 
Scientific  Review  Panel  and  Independent  Peer  Review69 

The  Independent  Scientific  Review  Panel,  established  by  a  collective  of  interdepartmental  and 
inter-governmental  offices,  is  to  be  “convened  by  a  body,  such  as  the  National  Academy  of 
Sciences,  to  review  the  plan’s  progress  toward  achieving  the  natural  system  restoration  goals  of 
the  Plan.”70  The  panel  is  to  produce  a  biennial  report  to  Congress  and  members  of  the  establishing 
body,  that  includes  an  assessment  of  the  ecological  indicators  and  other  measures  of  progress  in 
restoring  the  ecology  of  the  natural  system,  based  on  the  plan. 

In  addition,  the  enactment  calls  for  an  independent  peer  review  of  methods  for  project 
analysis.71  The  statute  charges  the  Secretary  of  the  Army  to  contract  with  the  National  Academy 
of  Sciences  (with  a  specified  authorization  of  appropriations),  to  conduct  a  study  that  includes 
reviews  of  various  methods  used  and  a  comparative  evaluation  of  the  basis  and  validity  of 
relevant  state-of-the-art  methods.  The  report — to  be  issued  not  later  than  one  year  after  the  date  of 
the  contract  and  submitted  to  the  Secretary  and  to  specified  congressional  committees — is  to 
include  the  results  of  the  study  and  specific  recommendations  for  modifying  any  methods  being 
used  in  the  project  for  conducting  economic  and  environmental  analyses  of  water  resource 
projects. 


Prince  William  Sound  (Alaska)  Oil  Spill  Recovery  Institute  and 
Advisory  Board72 

The  Oil  Spill  Recovery  Institute,  established  by  the  Secretary  of  Commerce,  is  to  conduct 
research  and  carry  out  educational  and  demonstration  projects  designed  to  develop  the  best 
available  techniques,  equipment,  and  materials  dealing  with  oil  spills.  In  addition,  the  institute  is 
to  complement  federal  and  state  damage  assessment  efforts  as  well  as  determine,  document, 
assess,  and  understand  the  long-range  effects  of  relevant  oil  spills.  The  institute  is  to  publish  and 
make  available  to  any  person  the  results  of  its  research,  educational,  and  demonstration  projects; 
copies  of  all  such  materials  are  to  be  provided  to  the  National  Oceanic  and  Atmospheric 
Administration. 

The  policies  of  the  institute  are  determined  by  an  Advisory  Board,  which  also  appoints  the 
institute’s  chair.  The  Advisory  Board,  itself  chaired  by  the  Secretary  of  Commerce,  consists  of 
federal  and  state  officials,  representatives  of  relevant  industries,  representatives  of  Alaska 
Natives,  and  residents  of  local  communities.  The  board  may  also  request  a  scientific  review  of  a 
research  program  it  authorized  every  five  years,  to  be  conducted  by  the  National  Academy  of 
Sciences.  The  board  shall  establish  a  Scientific  and  Technical  Committee,  composed  of  relevant 
specialists,  to  advise  and  make  recommendations  to  the  board  regarding  the  conduct  and  support 
of  research,  projects,  and  studies. 


69  P.L.  106-541;  114  Stat.  2595-2596  and  2691. 

70  114  Stat.  2691. 

71  114  Stat.  2595-2596. 

72  33  U.S.C.  2731. 
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Oil  Pollution  Research  Interagency  Coordinating  Committee73 

The  Oil  Pollution  Research  Interagency  Coordinating  Committee  includes  representatives 
from  a  number  of  federal  agencies.  It  is  chaired  by  the  representative  of  the  United  States  Coast 
Guard  and  operates  under  a  specified  budget.  The  committee  is  to  report  to  Congress  on  a  plan 
dealing  with  oil  pollution  research,  development,  and  demonstration  program.  At  this  initial  stage, 
the  enactment  requires  consultation  with  affected  states  on  certain  matters  and  contracting  with 
the  National  Academy  of  Sciences  for  advice  and  guidance  on  preparing  the  plan  and  assessing  its 
adequacy.  Subsequently,  the  committee  is  to  coordinate  the  establishment  of  a  program  for 
conducting  oil  pollution  research  and  development,  which  includes  requirements  for  monitoring 
and  evaluating  relevant  aspects  of  the  plan.  The  committee  is  also  authorized  to  make 
recommendations  for  grants  to  the  private  sector  to  an  appropriate  granting  agency  represented  on 
the  committee;  its  recommendations  are  to  ensure  an  appropriate  balance  within  a  region  among 
various  aspects  of  oil  pollution  research.  The  chairperson  of  the  committee  is  to  report  to 
Congress  every  two  years  on  its  activities  and  on  activities  proposed  and  carried  out  under  the  act. 

Water  Resources  Development  Independent  Peer  Review  Panels74 

Various  project  studies  authorized  by  the  act  are  subject  to  peer  review  by  independent  panels 
of  experts.  Some  reviews  are  mandatory,  while  others  are  discretionary;  the  latter  include  ones 
requested  by  a  governor  of  an  affected  state  or  certain  federal  agencies.  For  each  project  subject 
to  a  peer  review,  the  Chief  of  Engineers  of  the  Army  Corps  of  Engineers  is  to  contract  with  the 
National  Academy  of  Sciences  or  a  similar  independent  scientific  and  technical  advisory 
organization  to  establish  a  panel  of  experts,  who  are  to  represent  a  balance  of  areas  of  expertise 
suitable  for  the  review  and  who  have  no  conflict  or  direct  involvement  with  the  project  being 
reviewed.  Specifications  for  the  peer  review  include  assessing  the  adequacy  and  acceptability  of 
economic,  engineering,  and  environmental  methods,  models,  and  analyses. 

Certain  committees  of  Congress  are  to  be  notified  of  a  project  study  for  peer  review  prior  to  its 
initiation.  In  addition,  the  review  panels  are  to  submit  a  report  to  the  Chief  of  Engineers  not  more 
than  60  days  after  the  last  day  of  public  comment  for  a  draft  project  study  or  later,  if  the  chief 
determines  an  extension  is  necessary.  The  panels  are  also  directed  to  submit  to  the  chief  a  final 
report  containing  certain  analyses.  They  may  make  recommendations,  to  which  the  chief  shall 
issue  a  written  response  for  any  recommendation  adopted  or  not  adopted.  Copies  of  the  reports 
and  the  chief’s  responses  are  to  be  made  available  to  the  public  by  electronic  means  and 
transmitted  to  certain  congressional  committees.  The  enactment  authorizes  federal  funds  to  be 
used  for  each  panel;  it  imposes  a  ceiling  of  $500,000,  which  can  be  waived  by  the  chief  if  he  or 
she  determines  the  waiver  to  be  appropriate. 


73  33  U.S.C.  §  2761. 

74  33  U.S.C.  §  2343. 
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American  Recovery  and  Reinvestment  Act  (ARRA)  of  2009 
Recovery  Accountability  and  Transparency  Board  and  Independent 
Advisory  Panel75 

ARRA,  sometimes  referred  to  as  the  Recovery  Act,  provides  for  a  number  of  oversight 
mechanisms.  One  is  the  Recovery  Accountability  and  Transparency  Board,  established  “to 
coordinate  and  conduct  oversight  of  covered  funds  to  prevent  fraud,  waste,  and  abuse.”  The 
board’s  membership  includes  ten  specified  inspectors  general  and  “any  other  Inspector  General 
designated  by  the  President  from  any  agency  that  expends  or  obligates  covered  funds.”  The 
President  is  authorized  to  appoint  the  chairperson  of  the  board,  to  be  selected  from  among  three 
officers.  The  three  are:  the  OMB  deputy  director  for  management,  a  federal  officer  who  already 
has  been  confirmed  by  the  Senate  for  another  position,  or  an  individual  who  would  have  to  be 
confirmed  by  the  Senate.  (The  President  selected  the  sitting  Inspector  General  of  the  Department 
of  the  Interior  (a  confirmed  position)  to  be  chairperson.) 

The  Board  may  request  information  and  assistance  from  any  agency  or  entity  of  the  federal 
government.  Supplementing  this,  the  Board  is  given  the  same  powers  of  inspectors  general 
operating  under  the  IG  Act  of  1978,  as  amended.  It  may  conduct  public  hearings;  enter  into 
contracts  as  may  be  necessary  to  carry  out  its  duties;  and  transfer  funds  for  expenses  to  support 
administrative  services  and  audits,  reviews,  or  other  oversight  activities. 

The  Board  is  further  empowered  to  “conduct  its  own  independent  audits  and  reviews  relating  to 
covered  funds  and  collaborate  on  audits  and  reviews”  with  any  inspector  general.  In  addition,  the 
board’s  detailed  audit  and  review  functions  pertain  to  the  following:  covered  funds  for  relevant 
purposes;  the  reporting  and  competition  requirements  associated  with  contracts  and  grants  to 
determine  whether  these  meet  applicable  standards;  the  adequacy  and  training  of  personnel  to 
oversee  the  covered  funds;  and  the  extent  of  appropriate  mechanisms  for  interagency 
collaboration,  including  coordinating  and  collaborating  with  the  Council  of  the  Inspectors 
General  on  Integrity  and  Efficiency.  ARRA  also  directs  the  board  to  make  recommendations  to 
agencies  with  regard  to  preventing  waste,  fraud,  and  abuse  of  the  covered  funds.  An  agency  must 
respond  within  30  days  of  the  receipt  of  the  recommendations,  stating  whether  it  agrees  or 
disagrees  and  whether  any  actions  will  be  taken  to  implement  them. 

The  Board  is  also  authorized  to  issue  a  variety  of  reports  to  the  President  and  Congress.  These 
include  “flash  reports”  on  potential  management  and  funding  problems  that  require  immediate 
attention;  quarterly  reports  summarizing  the  findings  of  the  Board  and  inspectors  general  of 
agencies;  annual  reports  consolidating  the  quarterly  reports;  and  such  other  reports  that  the  Board 
deems  appropriate.  In  general,  all  of  these  reports  are  to  be  made  available  to  the  public  on  the 
Board’s  website.  The  website,  which  is  to  be  “user-friendly”  and  “public-facing,”  is  to  provide 
links  to  related  websites  as  well  as  materials  and  information  regarding  the  act  itself; 
accountability  matters,  including  findings  of  the  Board’s  and  IGs’  audits;  data  on  contracts  and 
grants  relating  to  covered  funds;  the  use  and  allocation  of  covered  funds  by  each  federal  agency; 
data  on  relevant  economic,  financial,  grant,  and  contract  information  to  enhance  public  awareness 
of  the  use  of  the  covered  funds;  and  to  the  extent  practical,  job  opportunities  afforded  by  the 
program  and  how  to  access  these  on  appropriate  federal,  state,  and  local  websites  and  locations. 


75  P.L.  111-5,  123  Stat.  289-296.  For  an  overview,  see  CRS  Report  R40572,  General  Oversight  Provisions  in  the 
American  Recovery  and  Reinvestment  Act  of 2009  (ARRA):  Requirements  and  Related  Issues,  by  Clinton  T.  Brass. 
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Supplementing  the  Board  is  a  Recovery  Independent  Advisory  Panel.  Also  established  by 
ARRA,  it  is  composed  of  five  members  appointed  by  the  President,  based  on  relevant  expertise. 
The  panel  is  to  “make  recommendations  to  the  Board  on  actions  the  Board  could  take  to  prevent 
waste,  fraud,  and  abuse  relating  to  covered  funds.”  The  panel  is  authorized  to  hold  hearings  and 
to  “secure  directly  from  any  agency  such  information  as  the  Panel  considers  necessary  to  carry 
out”  its  duties. 


Federal  Information  Security  Management  Act  (FISMA) 
Independent  Evaluations76 

FISMA  directs  that  “each  year,  each  agency  shall  have  performed  an  independent  evaluation  of 
the  information  security  program  and  practices  of  that  agency  to  determine  the  effectiveness  of 
such  program  and  practices.”77  The  Director  of  OMB — except  for  “national  security  systems” — is 
to  oversee  the  development  of  the  relevant  information  security  policies,  principles,  standards, 
and  guidelines;  review  and  (dis)approve  agency  security  programs;  and  take  certain  actions  to 
ensure  compliance  with  relevant  policies,  standards,  and  principles.78  Under  the  statute,  the 
Director — based  on  guidelines  and  standards  developed  by  the  National  Institute  of  Standards  and 
Technology  (NIST) — is  specifically  authorized  to  issue  information  security  guidelines  and 
standards,  including  minimum  standards,  which  “shall  be  compulsory  and  binding.”79 

The  statute  further  stipulates  that  the  “annual  evaluation  required  by  this  section  shall  be 
performed  by  the  Inspector  General  or  an  external  independent  auditor,”  as  determined  by 
the  Inspector  General  of  the  agency”  or  by  the  head  of  the  agency  if  an  IG  has  not  been 
established  by  law. 80  Such  an  independent  evaluation  is  to  include  (1)  a  testing  of  the 
effectiveness  of  information  security  control  techniques  and  (2)  an  assessment  (made  on  the  basis 
of  the  results  of  the  testing)  of  compliance  with  the  requirements. 81  These  annual  reviews  are  to 
include  the  evaluator’s  findings  and  recommendations,  which  are  communicated  to  the  agency 
head;  he  or  she  then  submits  the  results  of  the  reviews  to  the  Director  of  OMB,  who,  in  turn, 

“shall  summarize  the  results  of  the  evaluations”  and  report  these  to  Congress.82 


76  Codified  at  44  U.S.C.  Chapter  35.  FISMA  replaced  the  National  Defense  Authorization  Act  for  FY2001  (P.L.  106- 
398,  Title  X,  Subtitle  G,  sec.  1061;  1 14  Stat.  1654A-271-1654A-272),  which  had  already  contained  requirements  for 
annual  independent  evaluations  along  the  same  lines. 

77  44  U.S.C.  3545(a)(1). 

78  44  U.S.C.  3543(a)  and  3544(b).  Certain  national  security  systems,  however,  fall  under  the  jurisdiction  and  authority 
of  the  Director  of  National  Intelligence,  Director  of  the  Central  Intelligence  Agency,  and  Secretary  of  Defense.  44 
U.S.C.  3543(b)-(c)  and  3545(g)(3)  and  40  U.S.C.  1 1331(b)(2).  Another  provision  adds  that  “standards  and  guidelines 
for  national  security  systems  ....  shall  be  developed,  promulgated,  enforced,  and  overseen  as  otherwise  authorized  by 
law  and  as  directed  by  the  President.”  40  U.S.C.  1 1331(b)(1)(C).  An  earlier  version,  incidentally,  had  specified  that  the 
“President  may  disapprove  or  modify  the  standards  and  guidelines.”  40  U.S.C.  11331,  note  on  “Amendments.”  For 
additional  information,  see  U.S.  Department  of  Commerce,  NIST,  Information  Security:  Recommended  Security 
Controls  for  Federal  Information  Systems  and  Organizations,  NIST  Special  Publication  800-53,  Revision  3  (August 
2009).  The  NIST  report  recognizes  that  such  “standards  and  guidelines  shall  not  apply  to  national  security  systems 
without  the  express  approval  of  appropriate  federal  officials  exercising  policy  authority  over  such  systems.”  Ibid.,  p.  iii. 

79  44  U.S.C.  3543  (a)(3)  and  40  U.S.C.  1 1331(b)  and  (d);  and  NIST,  Information  Security,  with  the  exception  of 
national  security  information  systems,  as  previously  noted. 

80  44  U.S.C.  3545(b). 

81  44  U.S.C.  3545(a)(2)(A)-(B). 

82  44  U.S.C.  3545(g)  and  3543(a)(8). 
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Department  of  Defense  Independent  Study  on  Post-Traumatic 
Stress  Disorder  Efforts83 

Under  the  National  Defense  Authorization  Act  for  FY2010,  the  Secretary  of  Defense,  in 
consultation  with  the  Secretary  of  Veterans  Affairs,  is  to  provide  for  an  independent  study  on  the 
treatment  of  post-traumatic  stress  disorder.  The  study  is  to  be  “conducted  by  the  Institute  of 
Medicine  of  the  National  Academy  of  Sciences  or  such  other  independent  entity  as  the 
Secretary  shall  select  for  purposes  of  the  study.”84 

The  legislation  also  spells  out  certain  requirements  of  the  study.  These  include  a  listing  of  each 
operative  program  and  method  available  for  the  prevention,  screening,  diagnosis,  treatment,  or 
rehabilitation  of  the  disorder;  the  status  of  studies  and  clinical  trials  involving  innovative 
treatments;  a  description  of  each  treatment  program  and  a  comparison  of  methods  among  them,  at 
specified  locations;  current  and  projected  future  annual  expenditures  by  DOD  and  VAin  this 
matter;  and  a  description  of  gender-specific  and  racial  and  ethic  group-specific  mental  health 
treatment  and  services  available  for  members  of  the  Armed  Forces.  The  entity  conducting  the 
study  is  to  submit  an  initial  report,  due  on  July  1,  2012,  to  the  Secretaries  of  Defense  and  of 
Veterans  Affairs  and  appropriate  congressional  committees  (specifically,  the  House  and  Senate 
Committees  on  Ai  med  Services,  Appropriations,  and  Veterans’  Affairs;  the  House  Committee  on 
Energy  and  Commerce,  and  the  Senate  Committee  on  Health,  Education,  and  Welfare). 

Responses  by  the  Secretaries  to  the  report — including  any  recommendations  for  on  the  treatment 
of  the  disorder  based  on  the  report — are  required  six  months  later.  An  updated  report  and 
responses  are  to  be  submitted  by  July  1,  2014,  and  January  1,  2015,  respectively. 


Defense  Science  Board  Independent  Assessment  of  Improvements 
in  Service  Contracting85 

The  National  Defense  Authorization  Act  for  FY2010  provided  that  “the  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logistics  shall  direct  the  Defense  Science  Board  [DSB] 
to  conduct  an  independent  assessment  of  improvements  in  the  procurement  and  oversight  of 
services  by  the  Department  of  Defense.”86  The  assessment  is  to  cover  the  quality  and 
completeness  of  guidance  relating  to  the  procurement  of  services,  the  extent  to  which  best 
practices  are  being  developed  for  setting  requirements,  the  contracting  approaches  and  types  used 
for  the  procurement  of  services,  and  whether  effective  standards  to  measure  performance  have 
been  developed.  The  Under  Secretary,  who  receives  the  DSB  study,  is  “to  submit  to  the 


83  P.L.  1 11-84,  sec.  726  (123  Stat.  2395-2397). 

84  Ibid. 

85  P.L.  111-84,  sec.  802  (123  Stat.  2400-2401).  The  Defense  Science  Board  (DSB),  considered  a  federal  advisory 
commission,  was  established  by  the  Secretary  of  Defense  in  1956,  in  response  to  a  recommendation  of  the  Hoover 
Commission.  The  Board,  which  has  32  members  plus  seven  ex  officio  members,  operates  through  various  task  forces 
consisting  of  Board  members  and  other  experts/consultants,  to  conduct  studies  at  the  direction  of  various  DoD  officials. 
Under  its  charter,  the  DSB  is  to  provide  “independent  advice  and  recommendations  on  scientific,  technical, 
manufacturing,  acquisition  process,  and  other  matters  of  special  interest  to  the  Department  of  Defense.”  DSB  provides 
such  advice  to  the  Secretary  of  Defense  and  other  Department  officials  by  way  of  formal  briefings  and  written  reports. 
DOD.  Defense  Science  Board,  About  the  DSB,  History,  and  Charter:  Defense  Science  Board,  available  at 

ht  tp :// w  w  w.  acq .  osd.  mil/dsb/htm. 

86  P.L.  111-84,  sec.  802(b)  (123  Stat.  2400-2401). 
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congressional  defense  committees  a  report  on  the  results  of  the  assessment,  including  such 
comments  and  recommendations  as  the  Under  Secretary  considers  appropriate,”  by  a  specified 
date  (March  10,  2010,  following  the  law’s  enactment  on  October  28,  2009). 87 


National  Academy  of  Sciences  Review  of  National  Security 
Laboratories88 

The  National  Defense  Authorization  Act  for  FY2010  also  directs  the  Secretary  of  Energy  to 
“enter  into  an  agreement  with  the  National  Academy  of  Sciences  to  conduct  a  study  of’  three 
specified  laboratories.89  The  study  is  to  include  for  each  laboratory  an  evaluation:  of  the  quality  of 
the  scientific  research  and  the  engineering  being  conducted  there,  of  the  criteria  used  to  assess  the 
scientific  research  and  engineering,  of  the  relationship  between  the  quality  of  the  science  and 
engineering  and  the  contract  for  managing  and  operating  the  laboratory,  and  of  the  management 
of  work  conducted  by  the  laboratory  for  entities  other  than  the  Department  of  Energy.  The  NAS  is 
“to  submit  to  the  Secretary  of  Energy  a  report  containing  the  results  of  the  study  and  any 
recommendations  resulting  from  the  study.”90  By  a  specified  date  (January  1,  2011,  following  the 
October  28,  2009,  enactment),  the  Secretary  of  Energy  is  to  submit  to  the  appropriate  committees 
of  Congress  the  NAS  report  and  “any  comments  or  recommendations  of  the  Secretary  with 
respect  to  that  report.”91 


Federal  Transit  Administration  State  Safety  Oversight  Program  for 
Rail  Transit92 

The  Federal  Transit  Administration  (FTA),  a  part  of  the  Department  of  Transportation  (DOT), 
oversees  the  safety  and  security  of  rail  transit  agencies  which  operate  “rail  fixed  guideway  mass 
transportation  systems”  (e.g.,  metrorail  and  subways)  and  which  receive  federal  funds.97  Part  of 
the  process  includes  an  oversight  body  for  each  jurisdiction.  The  program  is  designed,  according 
to  a  GAO  summary,  “as  one  in  which  FTA,  other  federal  agencies,  states,  and  rail  transit  agencies 
collaborate  to  ensure  the  safety  and  security  of  rail  transit  systems.”94  In  most  cases,  each  rail 
transit  agency  exists  within  a  state,  although  a  few  extend  to  multi-state  and  other  jurisdictions  in 


87  Ibid.  Sec.  803(c)  (123  Stat.  2401). 

88  P.L.  111-84,  Sec  3131  (123  Stat.  2711-2712).  Under  Sec.  3131(c)  (123  Stat.  2711),  “the  Secretary  of  Energy  shall,  in 
consultation  with  the  Secretary  of  Defense  and  the  Director  of  National  Intelligence,  ensure  that  the  National  Academy 
of  Sciences  receives  full  and  timely  cooperation  from  the  Department  Energy,  Department  of  Defense,  and  the 
intelligence  community  ...  in  conducting  the  study.” 

89  Ibid. 

90  Ibid. 

91  Ibid. 

92  49  U.S.C.  5330  and  49  C.F.R.  659. 

93  49  U.S.C.  5330(a)-(b). 

94  GAO.  Rail  Transit:  Additional  Federal  Leadership  Would  Enhance  FTA ’s  State  Safety  Oversight  Program ,  GAO- 
06-821,  July  2006,  p.  3..  For  further  coverage,  see  CRS  Report  R40688,  The  Federal  Role  in  Rail  Transit  Safety,  by 
David  Randall  Peterman  and  William  J.  Mallett.  The  authors  recognize  that  "rail  transit  operations  are  an  inherently 
local  activity,  and  the  federal  government  has  limited  responsibility  for  the  safety  of  rail  transit  operations.”  Ibid., 
Summary. 
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such  locales  as  the  Washington,  DC  metropolitan  area  (which  includes  the  District  of  Columbia, 
Maryland,  and  Virginia).  Under  federal  law,  each  state  or  covered  area 

designates  a  State  authority  as  having  responsibility — (A)  to  require,  review,  approve,  and 
monitor  the  carrying  out  of  each  safety  plan;  (B)  to  investigate  hazardous  conditions  and 
accidents  on  the  systems;  and  (C)  to  require  corrective  action  to  correct  or  eliminate  those 
conditions  (49  U.S.C.  5330(c)). 

As  an  enforcement  mechanism,  the  DOT  Secretary  is  authorized  to  withhold  funds  if  a  state  does 
not  comply  with  these  responsibilities. 

The  FTA  program  requires  each  state  or  other  appropriate  jurisdictions  to  establish  an  oversight 
body  to  oversee  the  safety  and  security  of  its  system.  These  oversight  units  are  responsible  for 
developing  a  program  standard  that  transit  agencies  must  meet  and  for  reviewing  performance  of 
the  transit  agencies  against  that  standard.  The  oversight  agency  safety  and  security  review  lays 
out  the  process  and  criteria  to  be  used,  at  least  every  three  years,  in  conducting  a  complete  review 
of  each  affected  rail  transit  agency’s  implementation  of  the  plan. 

The  FTA  rail  safety  oversight  program,  operating  through  state  and  local  oversight  entities, 
involves 

•  several  federal  agencies,  including  components  in  the  Department  of  Homeland 
Security  and  others  in  DOT; 

•  jurisdiction  over  states  and  metropolitan  areas  and  agencies; 

•  oversight  units  that  are  separate  from  the  transit  operating  authorities; 

•  program  implementation  reviews  and  assessments;  and 

•  time  schedules  for  reporting. 

A  2006  GAO  report,  however,  found  deficiencies  in  the  system,  including  a  failure  to  meet  the 
three-year  review  schedule,  to  develop  performance  goals  for  the  program,  and  to  be  able  to  track 
performance.95  In  addition,  GAO  reported  that  expertise  varied  across  the  oversight  entities, 
which,  in  some  cases,  suffered  from  inadequacies  in  funding  and  qualified  staff.96 

Reinforcement  of  some  of  these  and  other  findings  came  about  after  a  serious  accident  on  the 
Washington  Metropolitan  Area  Transit  Authority  (WMATA)  Metrorail  system  in  mid-2009,  which 
revealed  short-comings  of  its  Tri-State  Oversight  Committee  (TOC).  The  Committee,  for 
instance,  lacked  its  own  e-mail  address,  mailing  address,  telephone  number;  more  importantly,  it 
had  limited  resources,  staff  skills,  and  expertise.  Additional  problems  included  ineffective  and 
inadequate  communications  with  the  WMATA  and  the  absence  of  a  process  to  evaluate  corrective 
action  plans  and  alternatives  to  TOC  proposals.97  These  defects  and  deficiencies,  in  turn, 


95  GAO.  Rail  Transit,  Highlights  and  p.  1. 

96  Ibid.  Additional  analysis  of  the  FTA  safety  program  is  in  CRS  Report  R40688,  The  Federal  Role  in  Rail  Transit 
Safety. 

97  The  FTA  findings  and  recommendations  along  with  the  TOC  response  appear  at  DOT,  FTA,  Tri-State  Oversight 
Committee  and  the  Washington  Metropolitan  Area  Transit  Authority:  Final  Audit  Report,  March  4.  2010  (Washington, 
DOT/FT  A,  2010);  and  TOC,  TOC  Response  to  FTA  Final  Audit  Report,  May  4,  2010  (Washington,  WMATA,  2010). 
For  additional  coverage,  see  U.S.  Congress,  House  Committee  on  Oversight  and  Government  Reform,  The  Washington 
Metro  System:  Safety,  Service,  and  Stability,  hearing,  111th  Cong.,  2nd  sess.,  April  21,  2010,  at 

(continued...) 
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generated  plans  from  the  administration  and  proposals  from  legislators  to  correct  them,  not  only 
in  the  TOC  but  throughout  the  federal  transit  safety  system. 


Forensic  Science  Study  Conducted  by  an  Independent  Forensic 
Science  Committee  Established  by  the  National  Academy  of 
Sciences" 

The  Science,  State,  Justice,  Commerce,  and  Related  Agencies  Appropriations  Act  of  2006 
authorized  the  “National  Academy  of  Sciences  [NAS]  to  conduct  a  study  of  forensic  science,  as 
described  in  the  Senate  report,”  accompanying  the  legislation. 100  The  Senate  report,  which 
authorized  NAS  to  establish  an  “independent  Forensic  Science  Committee,”  recognized  that 

there  exists  little  to  no  analysis  of  the  remaining  needs  of  the  [criminal  justice  and  law 
enforcement]  community  outside  of  the  area  of  DNA.  Therefore  ....  the  Committee  directs 
the  Attorney  General  to  provide  [funds]  to  the  National  Academy  of  Sciences  to  create  an 
independent  Forensic  Science  Committee.  This  Committee  shall  include  members  of  the 
forensics  community  representing  operational  crime  laboratories,  medical  examiners,  and 
coroners;  legal  experts;  and  other  scientists  deemed  appropriate.101 

The  Senate  report  then  set  forth  a  series  of  charges  to  the  Committee.  These  included  assess  the 
present  and  future  resource  needs  of  the  forensic  science  community,  make  recommendations  for 
maximizing  the  use  of  forensic  science  techniques,  identify  potential  scientific  advances,  make 
recommendations  for  programs  that  will  increase  the  number  of  qualified  forensic  scientists  and 
others  related  professions,  disseminate  best  practices  and  guidelines  concerning  the  collection  and 
analysis  of  forensic  evidence,  and  examine  the  role  of  forensic  science  in  homeland  security. 102 

The  NAS  follow-up  study  reported  on  the  members  of  the  Committee;  participants  in  hearings 
and  meetings;  literature  reviewed;  issues  covered;  and  findings,  conclusions,  and 
recommendations. 103 


(...continued) 

http://oversight.house.gov/indx.php?option=com_content&task=view&id=4892&Itemid=2. 

98  For  proposals  from  legislators  and  the  administration,  which,  in  part,  are  designed  to  improve  oversight,  see  National 
Metro  Safety  Act,  H.R.  3338,  and  S.  1506,  111th  Congress;  and  U.S.  Congress.  House  Committee  on  Transportation 
and  Infrastructure.  Subcommittee  on  Highways  and  Transit.  Public  Transit  Safety:  Examining  the  Federal  Role , 
hearing,  1 1 1th  Cong.,  1st  sess.,  December  8,  2009,  at  http://transportation.house.gov/hearings/hearingDetail.aspx? 
NewslD=1060. 

"P.L.  109-108,  119  Stat.  2290. 

100  Conference  Committee  report.  House  Rept.  109-272,  109th  Cong.,  1st  sess.,  p.  121.  The  NAS  study — Strengthening 
Forensic  Science  in  the  United  States:  A  Path  Forward — is  available  at  http://books.nap.edu/catagol/12598.html. 

101  Senate  rept.  109-88,  109th  Cong.,  1st  sess.,  p.  46. 

102  Ibid. 

103  NAS,  Strengthening  Forensic  Science,  pp.  1-7. 
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Improving  America's  Schools  Act  of  1994  Independent 
Evaluation104 

This  enactment  establishes  or  expands  on  a  number  of  educational  programs,  some  of  which  are 
required  to  undergo  an  independent  evaluation.  The  portion  of  the  act  dealing  with  the  “Even 
Start  Family  Literacy  Programs,”  for  instance,  directs  the  Secretary  of  Education  to 

provide  for  an  independent  evaluation  of  programs  assisted  under  this  part — (1)  to 
determine  the  performance  and  effectiveness  of  programs  assisted  under  this  part;  and  (2) 
identify  effective  Even  Start  programs  assisted  under  this  part  that  can  be  duplicated  and 
used  in  providing  technical  assistance  to  Federal,  State,  and  local  programs  (Sec.  1209). 


No  Child  Left  Behind  (NCLB)  Act  of  2001  Independent 
Evaluations,  Reviews,  and  Studies,  as  well  as  Peer  Reviews105 

The  NCLB  Act  includes  numerous  provisions  for  program  evaluations,  often  done  by  state 
agencies  involved  in  the  programs,  as  well  as  for  various  independent  evaluations,  reviews,  and 
studies.  In  some  of  these,  the  Secretary  of  Education: 

•  is  allocated  a  specified  funding  amount,  to  “conduct  an  independent  evaluation  of 
the  effectiveness”  of  the  Early  Reading  Program,  which  specifies  the  contents  of  an 
interim  and  a  final  report  (Sec.  1226). 

•  is  allocated  certain  funds  to  provide  for  an  independent  evaluation  of  Even  Start 
programs  for  several  designated  puiposes:  to  determine  the  performance  and 
effectiveness  of  them;  to  identify  effective  programs  that  can  be  duplicated  and  used  in 
providing  technical  assistance  to  federal,  state,  and  local  programs;  and  to  provide  state 
educational  agencies  and  relevant  entities  with  technical  assistance  to  ensure  that  local 
evaluations  provide  accurate  information  on  the  effectiveness  of  programs  (Sec.  1239). 

•  is  to  conduct  a  national  assessment  of  Title  I  (Improving  the  Academic  Achievement 
of  the  Disadvantaged),  adhering  to  an  extensive  set  of  criteria  and  standards,  with  the 
assistance  of  an  independent  review  panel.  It  is  composed  of  various  specialists, 
education  practitioners,  parents  and  school  board  members,  and  technical  experts,  with  a 
requirement  to  ensure  diversity  among  the  groups;  the  panel  is  to  consult  and  advise  the 
Secretary  on  methodological  and  other  issues,  including  adherence  to  the  highest 
standards  of  quality  with  respect  to  research  design,  statistical  analysis,  and  the 
dissemination  of  the  findings,  and  on  the  use  of  valid  and  reliable  measures  to  document 
program  implementation.  A  final  report  is  to  be  reviewed  by  two  independent  experts  in 
program  evaluation,  who  may  come  from  the  review  panel;  they  are  to  evaluate  and 
comment  on  the  degree  to  which  the  report  meets  the  criteria  and  standards  set  forth  and 
their  comments  are  to  be  transmitted  with  the  report  (Sec.  1501(d)). 

•  is  to  conduct  an  independent  study  of  assessments  used  for  state  accountability 
purposes  and  for  making  decisions  about  the  promotion  and  graduation  of  students. 
Components  of  the  study,  not  to  last  more  than  five  years,  are  identified;  and  its  purposes 


104  P.L.  103-382,  Sections  1202  and  1209;  108  Stat.  3578  and  3584. 

105  P.L.  107-110;  115  Stat.  1555,  1566,  1596-1597,  1890,  and  1931. 
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are  spelled  out:  to  synthesize  and  analyze  existing  research,  evaluate  academic 
assessment  and  accountability  systems  in  State  and  local  educational  agencies  and 
schools;  and  make  recommendations  to  the  Department  of  Education  and  specified 
congressional  committees.  The  Secretary  is  authorized  to  award  a  contract  for  the  study, 
through  a  peer  review  process,  to  an  organization  or  entity  capable  of  conducting 
rigorous,  independent  research  (Sec.  1503).  And 

•  is  directed  to  use  a  peer  review  process  to  review  whether  a  state  has  failed  to  make 
adequate  yearly  progress  for  a  specified  program  (Sec.  6162),  and  may  do  so  to  review 
certain  applications  (Sec.  7142). 


Agricultural  Research,  Extension,  and  Education  Reform  Act  of 
1998  Cooperative  State  Research,  Education,  and  Extension  Service 
Peer  and  Merit  Reviews  and  Advisory  Board  Review106 

This  1998  act  establishes  several  review  mechanisms  connected  with  various  grant  programs. 

One  calls  for  the  Secretary  of  the  Department  of  Agriculture  (USDA)  to  “establish  procedures 
that  provide  for  scientific  peer  review  of  each  agricultural  research  grant  administered  on  a 
competitive  basis,”  by  the  USDA  Cooperative  Extension  Service.  These  procedures  are  to  include 
“a  review  panel  [which]  shall  verify,  at  least  every  5  years,  that  each  research  activity  of  the 
Department  and  research  conducted  under  each  research  program  of  the  Department  has  scientific 
merit.”  The  enactment  directs  the  review  panel  to  consider  “the  scientific  merit  and  relevance  of 
the  activity  or  research  ....  and  the  multistate  significance  of  the  activity  or  research.”  To  help 
ensure  its  expertise  and  independence,  the  panel  is  to  be  “composed  of  individuals  with  scientific 
expertise,  a  majority  of  whom  are  not  employees  of  the  agency  whose  research  is  being 
reviewed,”  and  who,  to  the  maximum  extent  practicable,  are  to  be  selected  from  colleges  and 
universities. 

As  part  of  the  consideration  of  the  scientific  merit  for  each  research  activity,  the  enactment  sets 
up  procedures  for  a  “merit  review  of  each  agricultural  extension  or  education  grant  administered 
on  a  competitive  basis,”  by  the  Cooperative  Extension  Service.  To  be  eligible  for  a  grant,  relevant 
institutions  are  to  “establish  a  process  for  merit  review  of  the  activity  and  review  the  activity  in 
accordance  with  the  process.” 

The  results  of  the  panel  reviews  are  to  be  submitted  to  an  Advisory  Board,  which  is  to  review 
annually  the  relevance  of  priorities  for  such  programs  and  adequacy  of  funding  for  them.  The 
Secretary  is  required  to  consider  the  results  of  the  Advisory  Board’s  review  “when  formulating 
each  request  for  proposals,  and  evaluating  proposals”  involving  such  programs. 


106  P.L.  105-185;  112  Stat.  527-528. 
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Fund  for  the  Improvement  of  Post-Secondary  Education  (FIPSE) 
Independent  Outside  Evaluator107 

FIPSE,  operating  under  a  project  director,  provides  grants  connected  with  relevant  projects.  An 
independent  outside  evaluator  (IOE)  is  called  for  in  the  legislation.  Hired  by  the  project 
director,  the  IOE  “must  be  someone  who  does  not  stand  to  gain  personally  or  professionally  from 
the  project  results.”  According  to  the  department’s  guidance,  the  IOE  is  to  attend  the  project 
director’s  meeting  on  evaluating  a  FIPSE  grant  and  “to  assist  the  Director  in  completing  the 
initial  evaluation  plan/chart,  due  three  months  after  the  start  of  the  grant.”  The  IOE  is  also  to 
assist  the  director  in  a  number  of  matters,  including  offering  advice  about:  which  project 
objectives  would  lend  themselves  to  measurement  and  evaluation,  which  baseline  data  should  be 
collected,  which  measurement  instruments  could  be  used,  what  data  might  be  collected  from  a 
possible  comparison  or  control  group,  and  how  the  director  might  disseminate  evaluation  results 
to  interested  parties.  The  independent  outside  evaluator  is  also  to  assist  the  project  director  in 
designing  the  evaluation  instruments  and  to  write  the  evaluation  reports. 


The  Carl  D.  Perkins  Vocational  and  Technical  Education  Assistance 
Act  Independent  Advisory  Panel  and  Independent  Evaluation108 

This  legislation  creates  an  Independent  Advisory  Panel  (IAP) — appointed  by  the  Secretary  of 
Education  and  consisting  of  representatives  from  a  broad  range  of  experts  and  affected  parties — 
to  advise  him  or  her  on  the  implementation  and  assessment  of  the  programs  authorized  by  the  act. 
The  Secretary  is  also  authorized  to  collect  relevant  information  from  states  and  localities  which 
can  be  used  in  an  evaluation.  To  assist  in  this,  the  Secretary  “shall  provide  for  the  conduct  of  an 
independent  evaluation  and  assessment  of  vocational  and  technical  programs  under  this  Act 
through  studies  and  analyses  conducted  independently  through  grants,  contracts,  and  cooperative 
agreements  that  are  awarded  on  a  competitive  basis.”  The  enactment  also  details  the  contents  of 
such  an  assessment.  These  extend  to:  the  efforts  and  effects  of  state,  local,  and  tribal  entities  on 
such  programs;  impact  of  federal  expenditures  that  address  program  improvements  in  relevant 
educational  programs;  preparation  and  qualifications  of  teachers  in  the  fields;  academic  and 
employment  outcomes  of  the  education;  employer  involvement  and  satisfaction  with  such 
educational  programs;  use  and  impact  of  educational  technology  and  “distance  learning”  in  the 
field;  and  effect  of  state  adjusted  levels  of  performance  and  state  levels  of  performance  on  the 
delivery  of  relevant  services. 

The  Secretary  is  also  to  submit  to  specified  congressional  committees  an  interim  report  on  the 
assessment  and  a  final  report  summarizing  all  studies  and  analyses  related  to  the  assessment. 
These  reports  “shall  not  be  subject  to  any  review  outside  the  Department  of  Education”  prior  to 
their  submission  to  Congress.  But  "the  President,  Secretary,  and  advisory  panel ....  may  make 
such  additional  recommendations  to  Congress  with  respect  to  the  assessment”  as  each  determines 
appropriate. 


107  Department  of  Education,  Guidance  on  the  Role  of  the  Independent  Outside  Evaluator,  at  http://www.ed.gov/print/ 
about/offices/list/ope/fipse/outsideevaluator.html.  July  23,  2009. 

108  P.L.  105-332,  112  Stat.  3090-3093. 
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Occupational  Information  To  Be  Collected  by  the  Social  Security 
Administration  (SSA)  Independent  Evaluation109 

Under  this  highly  specialized,  narrowly  focused  request,  the  SSA  sought  the  services  of  an 
“Independent  Evaluator  (IE)”  regarding  “pre-award  and  post-award  evaluations  of 
occupational  information  and  methodology  employed  by  private  sector  entities.”  The  IE  was 
expected  to  help  revise  and  update  existing  coverage  published  by  the  Department  of  Labor.  Both 
pre-award  and  post-award  evaluation  services  and  requirements  are  detailed  in  the  SSA  request 
itself  and  in  its  attachments. 
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